spark mongo pipeline partitioning failed
ClassNotFoundException: ...MongoDefaultPartition
Partitioning using the 'DefaultMongoPartitioner$' failed.
java.lang.NoSuchMethodError: com.mongodb.client.MongoCollection.countDocuments(Lorg/bson/conversions/Bson;)
目前将partitioner设置为MongoSplitVectorPartitioner可解决
SparkConf().set("spark.mongodb.input.partitioner", "MongoSplitVectorPartitioner")
参考
https://docs.mongodb.com/spark-connector/current/configuration/index.html#input-configuration
no permission on database config
new SparkConf().set("spark.mongodb.input.uri", "mongodb://user:pwd@host:port/db.collection")
需要同时设置mongodb user的 config db read 权限
db.updateUser("readonly",{"roles":[{"role":"read","db":"yourdb"},{"role":"read","db":"config"}]});
