SolrCloud avec Docker Compose
Cet article sur la mise en place de SolrCloud fait écho à celui sur le cluster Cassandra avec Docker. Pour comprendre certains points abordés ci-dessous, il est nécessaire de connaitre la terminologie SolrCloud.
Premiers essais avec Docker
Comme précédemment, la première étape se résume à valider les images et la communication entre les conteneurs du cluster du moteur d’indexation via la ligne de commande docker. Comme souvent, l’équipe Docker propose une image officielle pour Solr avec un mode distribué qui peut être lancé de cette façon :
$ docker run --name myproject-zookeeper -d -p 2181:2181 -p 2888:2888 -p 3888:3888 jplock/zookeeper # démarrage de 4 instances Solr $ for i in `seq 1 4`; do docker run --name myproject-solr-$i --link myproject-zookeeper:ZK -d -p $(( 8982 + $i )):8983 solr bash -c '/opt/solr/bin/solr start -f -z $ZK_PORT_2181_TCP_ADDR:$ZK_PORT_2181_TCP_PORT'; done
Cette image contient également les outils pour créer une Collection. Pour cela, il faut l’exécuter directement sur l’un des conteneurs du cluster. Voici la commande pour créer une Collection découpée en 2 Shards eux-mêmes répliqués 2 fois :
$ docker exec -i -t myproject-solr-1 bin/solr create_collection -c myproject -shards 2 -replicationFactor 2 -p 8983
Connecting to ZooKeeper at 172.17.0.5:2181 ...
Uploading /opt/solr/server/solr/configsets/data_driven_schema_configs/conf for config myproject to ZooKeeper at 172.17.0.5:2181
Creating new collection 'myproject' using command:
http://localhost:8983/solr/admin/collections?action=CREATE&name=myproject&numShards=2&replicationFactor=2&maxShardsPerNode=1&collection.configName=myproject
{
"responseHeader":{
"status":0,
"QTime":8746},
"success":{"":{
"responseHeader":{
"status":0,
"QTime":8478},
"core":"myproject_shard1_replica2"}}}
En se rendant sur la console d’administration de SolrCloud (http://dockermachine:8983/solr/#/~cloud), il est possible de constater la création de la collection distribuée sur les 4 instances :
Il ne reste plus qu’à lancer l’indexation de documents toujours en utilisant les outils mis à disposition par Solr directement dans l’un des conteneurs du cluster :
$ docker exec -it --user=solr myproject-solr-1 bin/post -c myproject example/exampledocs/manufacturers.xml java -classpath /opt/solr/dist/solr-core-5.3.1.jar -Dauto=yes -Dc=myproject -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/manufacturers.xml SimplePostTool version 5.0.0 Posting files to [base] url http://localhost:8983/solr/myproject/update... Entering auto mode. File endings considered are xml,json,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log POSTing file manufacturers.xml (application/xml) to [base] 1 files indexed. COMMITting Solr index changes to http://localhost:8983/solr/myproject/update... Time spent: 0:00:01.150
Puis de vérifier la réplication sur chacun des noeuds :
for i in `seq 1 4`;
do
curl "http://dockermachine:$(( 8982 + $i ))/solr/myproject/select?q=*%3A*&wt=json"
done
{"responseHeader":{"status":0,"QTime":60,"params":{"q":"*:*","wt":"json"}},"response":{"numFound":11,"start":0,"maxScore":1.0,"docs":[{"id":"adata","compName_s":"A-Data Technology","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518135858246451200},{"id":"asus","compName_s":"ASUS Computer","address_s":"800 Corporate Way Fremont, CA 94539","_version_":1518135858502303744},{"id":"belkin","compName_s":"Belkin","address_s":"12045 E. Waterfront Drive Playa Vista, CA 90094","_version_":1518135858504400896},{"id":"maxtor","compName_s":"Maxtor Corporation","address_s":"920 Disc Drive Scotts Valley, CA 95066","_version_":1518135858511740928},{"id":"apple","compName_s":"Apple","address_s":"1 Infinite Way, Cupertino CA","_version_":1518135858281054208},{"id":"ati","compName_s":"ATI Technologies","address_s":"33 Commerce Valley Drive East Thornhill, ON L3T 7N6 Canada","_version_":1518135858631278592},{"id":"canon","compName_s":"Canon, Inc.","address_s":"One Canon Plaza Lake Success, NY 11042","_version_":1518135858640715776},{"id":"corsair","compName_s":"Corsair Microsystems","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518135858641764352},{"id":"dell","compName_s":"Dell, Inc.","address_s":"One Dell Way Round Rock, Texas 78682","_version_":1518135858641764353},{"id":"samsung","compName_s":"Samsung Electronics Co. Ltd.","address_s":"105 Challenger Rd. Ridgefield Park, NJ 07660-0511","_version_":1518135858642812928}]}}
{"responseHeader":{"status":0,"QTime":48,"params":{"q":"*:*","wt":"json"}},"response":{"numFound":11,"start":0,"maxScore":1.0,"docs":[{"id":"adata","compName_s":"A-Data Technology","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518135858246451200},{"id":"asus","compName_s":"ASUS Computer","address_s":"800 Corporate Way Fremont, CA 94539","_version_":1518135858502303744},{"id":"belkin","compName_s":"Belkin","address_s":"12045 E. Waterfront Drive Playa Vista, CA 90094","_version_":1518135858504400896},{"id":"maxtor","compName_s":"Maxtor Corporation","address_s":"920 Disc Drive Scotts Valley, CA 95066","_version_":1518135858511740928},{"id":"apple","compName_s":"Apple","address_s":"1 Infinite Way, Cupertino CA","_version_":1518135858281054208},{"id":"ati","compName_s":"ATI Technologies","address_s":"33 Commerce Valley Drive East Thornhill, ON L3T 7N6 Canada","_version_":1518135858631278592},{"id":"canon","compName_s":"Canon, Inc.","address_s":"One Canon Plaza Lake Success, NY 11042","_version_":1518135858640715776},{"id":"corsair","compName_s":"Corsair Microsystems","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518135858641764352},{"id":"dell","compName_s":"Dell, Inc.","address_s":"One Dell Way Round Rock, Texas 78682","_version_":1518135858641764353},{"id":"samsung","compName_s":"Samsung Electronics Co. Ltd.","address_s":"105 Challenger Rd. Ridgefield Park, NJ 07660-0511","_version_":1518135858642812928}]}}
{"responseHeader":{"status":0,"QTime":51,"params":{"q":"*:*","wt":"json"}},"response":{"numFound":11,"start":0,"maxScore":1.0,"docs":[{"id":"adata","compName_s":"A-Data Technology","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518135858246451200},{"id":"asus","compName_s":"ASUS Computer","address_s":"800 Corporate Way Fremont, CA 94539","_version_":1518135858502303744},{"id":"belkin","compName_s":"Belkin","address_s":"12045 E. Waterfront Drive Playa Vista, CA 90094","_version_":1518135858504400896},{"id":"maxtor","compName_s":"Maxtor Corporation","address_s":"920 Disc Drive Scotts Valley, CA 95066","_version_":1518135858511740928},{"id":"apple","compName_s":"Apple","address_s":"1 Infinite Way, Cupertino CA","_version_":1518135858281054208},{"id":"ati","compName_s":"ATI Technologies","address_s":"33 Commerce Valley Drive East Thornhill, ON L3T 7N6 Canada","_version_":1518135858631278592},{"id":"canon","compName_s":"Canon, Inc.","address_s":"One Canon Plaza Lake Success, NY 11042","_version_":1518135858640715776},{"id":"corsair","compName_s":"Corsair Microsystems","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518135858641764352},{"id":"dell","compName_s":"Dell, Inc.","address_s":"One Dell Way Round Rock, Texas 78682","_version_":1518135858641764353},{"id":"samsung","compName_s":"Samsung Electronics Co. Ltd.","address_s":"105 Challenger Rd. Ridgefield Park, NJ 07660-0511","_version_":1518135858642812928}]}}
{"responseHeader":{"status":0,"QTime":38,"params":{"q":"*:*","wt":"json"}},"response":{"numFound":11,"start":0,"maxScore":1.0,"docs":[{"id":"adata","compName_s":"A-Data Technology","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518135858246451200},{"id":"asus","compName_s":"ASUS Computer","address_s":"800 Corporate Way Fremont, CA 94539","_version_":1518135858502303744},{"id":"belkin","compName_s":"Belkin","address_s":"12045 E. Waterfront Drive Playa Vista, CA 90094","_version_":1518135858504400896},{"id":"maxtor","compName_s":"Maxtor Corporation","address_s":"920 Disc Drive Scotts Valley, CA 95066","_version_":1518135858511740928},{"id":"apple","compName_s":"Apple","address_s":"1 Infinite Way, Cupertino CA","_version_":1518135858281054208},{"id":"ati","compName_s":"ATI Technologies","address_s":"33 Commerce Valley Drive East Thornhill, ON L3T 7N6 Canada","_version_":1518135858631278592},{"id":"canon","compName_s":"Canon, Inc.","address_s":"One Canon Plaza Lake Success, NY 11042","_version_":1518135858640715776},{"id":"corsair","compName_s":"Corsair Microsystems","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518135858641764352},{"id":"dell","compName_s":"Dell, Inc.","address_s":"One Dell Way Round Rock, Texas 78682","_version_":1518135858641764353},{"id":"samsung","compName_s":"Samsung Electronics Co. Ltd.","address_s":"105 Challenger Rd. Ridgefield Park, NJ 07660-0511","_version_":1518135858642812928}]}}
Il a donc été possible de créer un cluster SolrCloud fonctionnel avec seulement 2 images Docker et 2 instructions bash.
Passage à Docker Compose
Deux commandes, c’est toujours trop, non ? Il est possible de décrire ce Solr distribué avec Docker Compose de la façon suivante :
zookeeper:
image: jplock/zookeeper
ports:
- "2181:2181"
- "2888:2888"
- "3888:3888"
solr:
image: solr
links:
- zookeeper
command: /opt/solr/bin/solr start -f -z zookeeper
Avec ce fichier, SolrCloud se démarre avec une seule commande :
$ docker-compose up -d Starting dockermyproject_zookeeper_1 Starting dockermyproject_solr_1
Il est toujours possible de scaler le cluster :
$ docker-compose scale solr=4 Creating and starting 2 ... done Creating and starting 3 ... Creating and starting 3 ... done Creating and starting 4 ... done
La création de la collection se fait toujours de la même façon :
$ docker exec -it dockermyproject_solr_1 bin/solr create_collection -c myproject -shards 2 -replicationFactor 2
Connecting to ZooKeeper at zookeeper ...
Uploading /opt/solr/server/solr/configsets/data_driven_schema_configs/conf for config myproject to ZooKeeper at zookeeper
Creating new collection 'myproject' using command:
http://localhost:8983/solr/admin/collections?action=CREATE&name=myproject&numShards=2&replicationFactor=2&maxShardsPerNode=1&collection.configName=myproject
{
"responseHeader":{
"status":0,
"QTime":18714},
"success":{"":{
"responseHeader":{
"status":0,
"QTime":17826},
"core":"myproject_shard2_replica1"}}}
En revanche l’indexation de données pose un souci d’accès au cluster:
$ docker exec -it dockermyproject_solr_1 bin/post -c myproject example/exampledocs/manufacturers.xml
java -classpath /opt/solr/dist/solr-core-5.3.1.jar -Dauto=yes -Dc=myproject -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/manufacturers.xml
SimplePostTool version 5.0.0
Posting files to [base] url http://localhost:8983/solr/myproject/update...
Entering auto mode. File endings considered are xml,json,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file manufacturers.xml (application/xml) to [base]
SimplePostTool: WARNING: Solr returned an error #500 ({msg=Error trying to proxy request for url: http://172.17.0.90:8983/solr/myproject/update,trace=org.apache.solr.common.SolrException: Error trying to proxy request for url: http://172.17.0.90:8983/solr/myproject/update? at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:598)? at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:446)? at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)? at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)? at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)? at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)? at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)? at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)? at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)? at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextH) for url: http://localhost:8983/solr/myproject/update
SimplePostTool: WARNING: Response: <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 {msg=Error trying to proxy request for url: http://172.17.0.90:8983/solr/myproject/update,trace=org.apache.solr.common.SolrException: Error trying to proxy request for url: http://172.17.0.90:8983/solr/myproject/update
at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:598)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:446)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:565)
... 23 more
,code=500}</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /solr/myproject/update. Reason:
<pre> {msg=Error trying to proxy request for url: http://172.17.0.90:8983/solr/myproject/update,trace=org.apache.solr.common.SolrException: Error trying to proxy request for url: http://172.17.0.90:8983/solr/myproject/update
at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:598)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:446)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:565)
... 23 more
,code=500}</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>
SimplePostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8983/solr/myproject/update
1 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/myproject/update...
SimplePostTool: WARNING: Solr returned an error #500 ({msg=Error trying to proxy request for url: http://172.17.0.91:8983/solr/myproject/update,trace=org.apache.solr.common.SolrException: Error trying to proxy request for url: http://172.17.0.91:8983/solr/myproject/update? at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:598)? at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:446)? at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)? at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)? at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)? at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)? at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)? at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)? at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)? at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextH) for url: http://localhost:8983/solr/myproject/update?commit=true
SimplePostTool: WARNING: Response: <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 {msg=Error trying to proxy request for url: http://172.17.0.91:8983/solr/myproject/update,trace=org.apache.solr.common.SolrException: Error trying to proxy request for url: http://172.17.0.91:8983/solr/myproject/update
at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:598)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:446)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:565)
... 23 more
,code=500}</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /solr/myproject/update. Reason:
<pre> {msg=Error trying to proxy request for url: http://172.17.0.91:8983/solr/myproject/update,trace=org.apache.solr.common.SolrException: Error trying to proxy request for url: http://172.17.0.91:8983/solr/myproject/update
at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:598)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:446)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:565)
... 23 more
,code=500}</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>
Time spent: 0:00:24.074
Que ce passe-t-il ? Le port d’écoute de Solr n’a pas été rendu disponible depuis « l’extérieur ».
Une autre façon de décrire un SolrCloud distribué peut être envisagée :
zookeeper:
image: jplock/zookeeper
ports:
- "2181:2181"
- "2888:2888"
- "3888:3888"
solr0:
image: solr
links:
- zookeeper
ports:
- "8983:8983"
command: /opt/solr/bin/solr start -f -z zookeeper
solrn:
image: solr
links:
- zookeeper
command: /opt/solr/bin/solr start -f -z zookeeper
Voici le nouvel essai de création et scalabilité du cluster :
$ docker-compose up -d Starting dockermyproject_zookeeper_1 Starting dockermyproject_solr0_1 Creating dockermyproject_solrn_1 $ docker-compose scale solrn=3 Creating and starting 2 ... done Creating and starting 3 ... Creating and starting 3 ... done
La création de la collection est assez classique :
$ docker exec -it dockermyproject_solr0_1 bin/solr create_collection -c myproject -shards 2 -replicationFactor 2
Connecting to ZooKeeper at zookeeper ...
Uploading /opt/solr/server/solr/configsets/data_driven_schema_configs/conf for config myproject to ZooKeeper at zookeeper
Creating new collection 'myproject' using command:
http://localhost:8983/solr/admin/collections?action=CREATE&name=myproject&numShards=2&replicationFactor=2&maxShardsPerNode=1&collection.configName=myproject
{
"responseHeader":{
"status":0,
"QTime":22830},
"success":{"":{
"responseHeader":{
"status":0,
"QTime":21188},
"core":"myproject_shard2_replica2"}}}
Cette fois d’indexation des données fonctionne :
$ docker exec -it dockermyproject_solr0_1 bin/post -c myproject example/exampledocs/manufacturers.xml java -classpath /opt/solr/dist/solr-core-5.3.1.jar -Dauto=yes -Dc=myproject -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/manufacturers.xml SimplePostTool version 5.0.0 Posting files to [base] url http://localhost:8983/solr/myproject/update... Entering auto mode. File endings considered are xml,json,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log POSTing file manufacturers.xml (application/xml) to [base] 1 files indexed. COMMITting Solr index changes to http://localhost:8983/solr/myproject/update... Time spent: 0:00:01.236
Il est possible de vérifier la bonne réplication sur les autres noeuds du cluster :
for i in `seq 1 3`;
do
docker exec -it dockermyproject_solrn_$i wget -q -O- 'http://localhost:8983/solr/myproject/select?q=*%3A*&wt=json'
done
{"responseHeader":{"status":0,"QTime":51,"params":{"q":"*:*","wt":"json"}},"response":{"numFound":11,"start":0,"maxScore":1.0,"docs":[{"id":"adata","compName_s":"A-Data Technology","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518137560790990848},{"id":"asus","compName_s":"ASUS Computer","address_s":"800 Corporate Way Fremont, CA 94539","_version_":1518137560893751296},{"id":"belkin","compName_s":"Belkin","address_s":"12045 E. Waterfront Drive Playa Vista, CA 90094","_version_":1518137560897945600},{"id":"maxtor","compName_s":"Maxtor Corporation","address_s":"920 Disc Drive Scotts Valley, CA 95066","_version_":1518137560918917120},{"id":"apple","compName_s":"Apple","address_s":"1 Infinite Way, Cupertino CA","_version_":1518137560958763008},{"id":"ati","compName_s":"ATI Technologies","address_s":"33 Commerce Valley Drive East Thornhill, ON L3T 7N6 Canada","_version_":1518137561087737856},{"id":"canon","compName_s":"Canon, Inc.","address_s":"One Canon Plaza Lake Success, NY 11042","_version_":1518137561100320768},{"id":"corsair","compName_s":"Corsair Microsystems","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518137561113952256},{"id":"dell","compName_s":"Dell, Inc.","address_s":"One Dell Way Round Rock, Texas 78682","_version_":1518137561132826624},{"id":"samsung","compName_s":"Samsung Electronics Co. Ltd.","address_s":"105 Challenger Rd. Ridgefield Park, NJ 07660-0511","_version_":1518137561141215232}]}}
{"responseHeader":{"status":0,"QTime":70,"params":{"q":"*:*","wt":"json"}},"response":{"numFound":11,"start":0,"maxScore":1.0,"docs":[{"id":"adata","compName_s":"A-Data Technology","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518137560790990848},{"id":"asus","compName_s":"ASUS Computer","address_s":"800 Corporate Way Fremont, CA 94539","_version_":1518137560893751296},{"id":"belkin","compName_s":"Belkin","address_s":"12045 E. Waterfront Drive Playa Vista, CA 90094","_version_":1518137560897945600},{"id":"maxtor","compName_s":"Maxtor Corporation","address_s":"920 Disc Drive Scotts Valley, CA 95066","_version_":1518137560918917120},{"id":"apple","compName_s":"Apple","address_s":"1 Infinite Way, Cupertino CA","_version_":1518137560958763008},{"id":"ati","compName_s":"ATI Technologies","address_s":"33 Commerce Valley Drive East Thornhill, ON L3T 7N6 Canada","_version_":1518137561087737856},{"id":"canon","compName_s":"Canon, Inc.","address_s":"One Canon Plaza Lake Success, NY 11042","_version_":1518137561100320768},{"id":"corsair","compName_s":"Corsair Microsystems","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518137561113952256},{"id":"dell","compName_s":"Dell, Inc.","address_s":"One Dell Way Round Rock, Texas 78682","_version_":1518137561132826624},{"id":"samsung","compName_s":"Samsung Electronics Co. Ltd.","address_s":"105 Challenger Rd. Ridgefield Park, NJ 07660-0511","_version_":1518137561141215232}]}}
{"responseHeader":{"status":0,"QTime":64,"params":{"q":"*:*","wt":"json"}},"response":{"numFound":11,"start":0,"maxScore":1.0,"docs":[{"id":"adata","compName_s":"A-Data Technology","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518137560790990848},{"id":"asus","compName_s":"ASUS Computer","address_s":"800 Corporate Way Fremont, CA 94539","_version_":1518137560893751296},{"id":"belkin","compName_s":"Belkin","address_s":"12045 E. Waterfront Drive Playa Vista, CA 90094","_version_":1518137560897945600},{"id":"maxtor","compName_s":"Maxtor Corporation","address_s":"920 Disc Drive Scotts Valley, CA 95066","_version_":1518137560918917120},{"id":"apple","compName_s":"Apple","address_s":"1 Infinite Way, Cupertino CA","_version_":1518137560958763008},{"id":"ati","compName_s":"ATI Technologies","address_s":"33 Commerce Valley Drive East Thornhill, ON L3T 7N6 Canada","_version_":1518137561087737856},{"id":"canon","compName_s":"Canon, Inc.","address_s":"One Canon Plaza Lake Success, NY 11042","_version_":1518137561100320768},{"id":"corsair","compName_s":"Corsair Microsystems","address_s":"46221 Landing Parkway Fremont, CA 94538","_version_":1518137561113952256},{"id":"dell","compName_s":"Dell, Inc.","address_s":"One Dell Way Round Rock, Texas 78682","_version_":1518137561132826624},{"id":"samsung","compName_s":"Samsung Electronics Co. Ltd.","address_s":"105 Challenger Rd. Ridgefield Park, NJ 07660-0511","_version_":1518137561141215232}]}}
Conclusion
Encore une fois, Docker Compose permet de monter rapidement et très simplement un environnement distribué. Il faut savoir que même si Zookeeper est une solution connue pour faire du Service Discovery dans un environnement Docker, ce n’est pas la solution la plus récommandée car la mise en place peut être complexe. Dans cet exemple, SolrCloud implémente lui-même l’intégration avec Zookeeper. Il n’y donc pas de sur-coût en terme d’implémentation.
Après avoir intégré un cluster de bases de données NoSQL et un moteur d’indexation distribué, quelle sera la prochaine étape ? 😛
https://blog.lecacheur.com/2015/11/18/solrcloud-avec-docker-compose/https://blog.lecacheur.com/wp-content/uploads/2015/11/Solr_Logo_on_white-1024x517.pnghttps://blog.lecacheur.com/wp-content/uploads/2015/11/Solr_Logo_on_white-150x150.pngDéveloppementLinuxcluster,docker,solrCet article sur la mise en place de SolrCloud fait écho à celui sur le cluster Cassandra avec Docker. Pour comprendre certains points abordés ci-dessous, il est nécessaire de connaitre la terminologie SolrCloud. Premiers essais avec Docker Comme précédemment, la première étape se résume à valider les images et la communication...SeBSébastien LECACHEUR23r0@laposte.netAdministratorLe weblogue de SeB
Thanks very good article, but I can not find the zkServer.sh, it is not in git 🙁