Error with index_cluster_stats.py

[taoran34]

I'm following the instructions for the QuickStart.

When it comes to step 6 with the command " python index_cluster_stats.py ", it fails with the following. I am confused with the port 9026, it just shows up suddenly, I do not know when it get configured. Should I change the port?

ERROR:inviso.cluster:Error processing: cluster_1
ERROR:inviso.cluster:HTTPConnectionPool(host='taoran1', port=9026): Max retries exceeded with url: /ws/v1/cluster/apps?state=RUNNING (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
Traceback (most recent call last):
File "index_cluster_stats.py", line 90, in index_stats
index_apps(es, cluster, info)
File "index_cluster_stats.py", line 17, in index_apps
apps = requests.get('http://%s:%s/ws/v1/cluster/apps?state=RUNNING' % (cluster.host, '9026'), headers = {'ACCEPT':'application/json'}).json().get('apps')
File "/home/venv/lib/python2.6/site-packages/requests/api.py", line 55, in get
return request('get', url, *_kwargs)
File "/home/venv/lib/python2.6/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, *_kwargs)
File "/home/venv/lib/python2.6/site-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, *_send_kwargs)
File "/home/venv/lib/python2.6/site-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, *_kwargs)
File "/home/venv/lib/python2.6/site-packages/requests/adapters.py", line 375, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='taoran1', port=9026): Max retries exceeded with url: /ws/v1/cluster/apps?state=RUNNING (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

Also, when conducted the command "python jes.py", it came up with the followings, I do not know whether it matters for the Error above.

INFO:inviso-monitor:Publishing event: (cluster_1) job_1413958950234_0008 2014-10-23T02:37:35+00:00
INFO:inviso-handler:Processing 1 events
WARNING:inviso-handler:No trace info available for hdfs://taoran1:9000/tmp/hadoop-yarn/staging/history/done/2014/10/23/000000/job_1413958950234_0008-1414031808636-hadoop-QuasiMonteCarlo-1414031855566-4-1-SUCCEEDED-default-1414031818339.jhist
INFO:inviso-handler:Indexing 1 documents
INFO:inviso-handler:Events complete.

[danielcweeks]

The default ports are based on EMR, which is not the hadoop default.

Just change the ports in those scripts to whatever you're running your RM on.

[taoran34]

Thanks a lot! I have changed the ports and it works.
But it does not work so well when I search for job information.

I search a user's job information, but the "Stop", "Duration", "Links", "Workflow ID" and "Genie Name" information is not showing up. Moreover, the "Job Status" shows "UNKNOWN".

Dose it have any connection with my hadoop configuration?
Which file finish the work of gathering jobs' information?

[danielcweeks]

So, that means the the job history isn't loading properly and just the job config information is available. You can check the logs from the tomcat server (catalina.out and inviso.log) to see if there are exceptions.

[taoran34]

Thanks a lot!
And does it matter whether the commands in QuickStart conducted as "root" or "hadoop" ?

[taoran34]

I have checked the catalina.out, and there is an exception as followings:

严重: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=EXECUTE, inode="/tmp":hadoop:supergroup:drwx------
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:205)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:168)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:5441)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1707)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1659)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1639)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1613)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:497)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:322)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
    at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1144)
    at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1132)
    at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1122)
    at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
    at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
    at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
    at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1295)
    at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
    at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
    at com.netflix.bdp.inviso.history.TraceService.trace(TraceService.java:133)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
    at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
    at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.google.inject.servlet.DefaultFilterPipeline.dispatch(DefaultFilterPipeline.java:43)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

[danielcweeks]

It looks like you're running tomcat as root instead of as hadoop

user=root, access=EXECUTE, inode="/tmp":hadoop:supergroup:drwx------

You can either run tomcat as hadoop or update your tomcat/bin/setenv.sh to export the hadoop user as hadoop:

export HADOOP_USER_NAME=hadoop

This will give the REST service access to read the files from hdfs.

[taoran34]

Thanks a lot!

[Slouis61]

In step 6 when I try and complete the process at:

python jes.py

I get:
ERROR:inviso.jes:[Errno 111] Connection refused
Traceback (most recent call last):
File "jes.py", line 35, in main
monitor.run()

File "/inviso/jes/inviso/monitor.py", line 295, in run for f in listing:

File "/venv/lib/python2.6/site-packages/snakebite/client.py", line 139, in ls recurse=recurse):

File "/venv/lib/python2.6/site-packages/snakebite/client.py", line 1072, in _find_items fileinfo = self._get_file_info(path)

File "/venv/lib/python2.6/site-packages/snakebite/client.py", line 1202, in _get_file_info return self.service.getFileInfo(request)

File "/venv/lib/python2.6/site-packages/snakebite/service.py", line 35, in rpc = lambda request, service=self, method=method.name: service.call(service_stub_class.dict[method], request)

File "/venv/lib/python2.6/site-packages/snakebite/service.py", line 41, in call return method(self.service, controller, request)

File "/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py", line 267, in self._StubMethod(inst, method, rpc_controller, request, callback))

File "/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py", line 284, in _StubMethod method_descriptor.output_type._concrete_class, callback)

File "/venv/lib/python2.6/site-packages/snakebite/channel.py", line 409, in CallMethod self.get_connection(self.host, self.port)

File "/venv/lib/python2.6/site-packages/snakebite/channel.py", line 211, in get_connection self.sock.connect((host, port))

File "", line 1, in connect

error: [Errno 111] Connection refused
(venv)

Any help would be greatly appreciated.. we Use Ambari.. running on port 8080 and Horton if this makes any difference. I would think this would affect the default port used by inviso?

[danielcweeks]

It looks like the same port issue most people are having. The default port used by inviso is the port used by EMR. Take a look at the ports used by your RM and update they python script accordingly (These probably need to be pulled out into the settings file).

[Slouis61]

Daniel, Thanks so much for the prompt reply! What is interesting is that even after I reinstalled the entire package and moved it to the resource manager node locally I still get the same problem.. I also moved the port to 10004 instead of 8080 and moved 9026 to what Horton uses 8088. I added the port=8020 as you specify above.
Any suggestions would be appreciated.
Still, I get:

)[root@bcwnode10 jes]# python jes.py
ERROR:inviso.jes:[Errno 111] Connection refused
Traceback (most recent call last):
File "jes.py", line 36, in main
monitor.run()
File "/opt/inviso/inviso/jes/inviso/monitor.py", line 295, in run
for f in listing:
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/client.py", line 139, in ls
recurse=recurse):
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/client.py", line 1072, in _find_items
fileinfo = self._get_file_info(path)
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/client.py", line 1202, in _get_file_info
return self.service.getFileInfo(request)
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/service.py", line 35, in
rpc = lambda request, service=self, method=method.name: service.call(service_stub_class.dict[method], request)
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/service.py", line 41, in call
return method(self.service, controller, request)
File "/opt/inviso/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py", line 267, in
self._StubMethod(inst, method, rpc_controller, request, callback))
File "/opt/inviso/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py", line 284, in _StubMethod
method_descriptor.output_type._concrete_class, callback)
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/channel.py", line 409, in CallMethod
self.get_connection(self.host, self.port)
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/channel.py", line 211, in get_connection
self.sock.connect((host, port))
File "", line 1, in connect
error: [Errno 111] Connection refused
(venv)[root@bcwnode10 jes]#

[danielcweeks]

Hmm, can you make sure you the NN is listening on localhost?

Try a 'telnet localhost 8020'. Otherwise specify host or ip in settings.py.

From the trace, I can't tell if it can't talk to the NameNode or the
DataNode.

Also take a look at this fork here(probably not current the master), which
might have the same settings:
https://github.com/mikebin/inviso/commit/ae822d4eb8b4b87574b412df7c15ee9e71619c5f

Let me know if any of that helps.

On Thu, Feb 5, 2015 at 8:52 AM, Slouis61 notifications@github.com wrote:

Daniel, Thanks so much for the prompt reply! What is interesting is that
even after I reinstalled the entire package and moved it to the resource
manager node locally I still get the same problem.. I also moved the port
to 10004 instead of 8080 and moved 9026 to what Horton uses 8088. I added
the port=8020 as you specify above.
Any suggestions would be appreciated.
Still, I get:

)[root@bcwnode10 jes]# python jes.py
ERROR:inviso.jes:[Errno 111] Connection refused
Traceback (most recent call last):
File "jes.py", line 36, in main
monitor.run()
File "/opt/inviso/inviso/jes/inviso/monitor.py", line 295, in run
for f in listing:
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/client.py",
line 139, in ls
recurse=recurse):
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/client.py",
line 1072, in

_find_items fileinfo = self.get_file_info(path) File
"/opt/inviso/venv/lib/python2.6/site-packages/snakebite/client.py", line
1202, in _get_file_info return self.service.getFileInfo(request) File
"/opt/inviso/venv/lib/python2.6/site-packages/snakebite/service.py", line
35, in rpc = lambda request, service=self, method=method.name
http://method.name: service.call(service_stub_class._dict[method],
request)
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/service.py",
line 41, in call
return method(self.service, controller, request)
File
"/opt/inviso/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py",
line 267, in
self._StubMethod(inst, method, rpc_controller, request, callback))
File
"/opt/inviso/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py",
line 284, in _StubMethod
method_descriptor.output_type._concrete_class, callback)
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/channel.py",
line 409, in CallMethod
self.get_connection(self.host, self.port)
File "/opt/inviso/venv/lib/python2.6/site-packages/snakebite/channel.py",
line 211, in get_connection
self.sock.connect((host, port))
File "", line 1, in connect
error: [Errno 111] Connection refused
(venv)[root@bcwnode10 jes]#

—
Reply to this email directly or view it on GitHub
#3 (comment).

[Slouis61]

Unfortunately no it did not work.. I will keep trying.. We love the promise of inviso hoping we can get it working with HDP 2.2. Thanks for your help. if you have any other ideas let me know.

[Slouis61]

and yes I can telent to the host IP in the settings.py and used an FQDN:
[root@bcwnode10 jes]# telnet bcwnode1.buzzanglepro.com 8020
Trying 10.0.5.130...
Connected to bcwnode1.buzzanglepro.com.

[danielcweeks]

@Slouis61 I just pulled down the HDP 2.2 Sandbox and was able to get it all running with the following mods:

jes.py

--- a/jes/jes.py
+++ b/jes/jes.py
@@ -27,6 +27,7 @@ def main():
                                           cluster_id=cluster.id,
                                           cluster_name=cluster.name,
                                           host=cluster.host,
+                                          port=8020,
                                           publisher=publisher,
                                           elasticsearch=settings.elasticsearch))

--- a/jes/inviso/monitor.py
+++ b/jes/inviso/monitor.py
@@ -277,7 +277,7 @@ class HdfsMr2LogMonitor(ElasticSearchMonitor):
                  cluster_name,
                  host='localhost',
                  port=9000,
-                 log_path='/tmp/hadoop-yarn/staging/history/done', **kwargs):
+                 log_path='/mr-history/done', **kwargs):
         super(HdfsMr2LogMonitor, self).__init__(**kwargs)


--- a/jes/index_cluster_stats.py
+++ b/jes/index_cluster_stats.py
@@ -14,7 +14,7 @@ log = get_logger('inviso.cluster')
 es_index = 'inviso-cluster'

 def index_apps(es, cluster, info):
-    apps = requests.get('http://%s:%s/ws/v1/cluster/apps?state=RUNNING' % (cluster.host, '9026'), headers = {'ACCEPT':'application/json'}).jso
+    apps = requests.get('http://%s:%s/ws/v1/cluster/apps?state=RUNNING' % (cluster.host, '8088'), headers = {'ACCEPT':'application/json'}).jso

     if not apps:
         log.info(cluster.name + ': no applications running.')
@@ -51,7 +51,7 @@ def index_apps(es, cluster, info):
     log.debug(bulk(es, documents, stats_only=True));

 def index_metrics(es, cluster, info):
-    metrics = requests.get('http://%s:%s/ws/v1/cluster/metrics' % (cluster.host, '9026'), headers = {'ACCEPT':'application/json'}).json()['clu
+    metrics = requests.get('http://%s:%s/ws/v1/cluster/metrics' % (cluster.host, '8088'), headers = {'ACCEPT':'application/json'}).json()['clu
     metrics.update(info)

     r = es.index(index=es_index,
@@ -63,7 +63,7 @@ def index_metrics(es, cluster, info):
     log.debug(r)

 def index_scheduler(es, cluster, info):
-    scheduler = requests.get('http://%s:%s/ws/v1/cluster/scheduler' % (cluster.host, '9026'), headers = {'ACCEPT':'application/json'}).json()[
+    scheduler = requests.get('http://%s:%s/ws/v1/cluster/scheduler' % (cluster.host, '8088'), headers = {'ACCEPT':'application/json'}).json()[
     scheduler.update(info)

[Slouis61]

it seems that HDP 2.2 sets a different user for hadoop. Once set to the proper user and the proper fqdn's used with the proper ports, all went through with out a hitch.

The only issue i am having now is with the public/UI. Its a fresh install of Apache Tomcat 7
Version 7.0.57.. My java version is 1.7.0_45; I get 2 screens:

and on refresh I get:

it looks like a servlet issue?

I will add some comments to assist others with getting it running and what needs to be tweaked on non sandbox implementations as soon as it is all operational.

Again THANKS!!!