2023-02-19 00:57:26 [0ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10227] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10227] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10227] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10227] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10227] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10227] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10227] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10227] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10227] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10227] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 00:57:26 [3ms] [Info] [omni.server] [10227] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "" } } } 2023-02-19 00:57:26 [3ms] [Info] [carb] [10227] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 00:57:26 [3ms] [Info] [carb] [10227] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 00:57:26 [4ms] [Info] [carb] [10227] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 00:57:26 [4ms] [Info] [carb] [10227] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 00:57:26 [4ms] [Info] [carb.launcher.plugin] [10227] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x16c13b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 00:57:26 [4ms] [Info] [carb.launcher.plugin] [10227] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 10227. 2023-02-19 00:57:26 [4ms] [Info] [carb] [10227] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 00:57:26 [4ms] [Info] [carb] [10227] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 00:57:26 [5ms] [Info] [carb.launcher.plugin] [10253] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 10254 from the parent 10227. 2023-02-19 00:57:26 [5ms] [Info] [carb.launcher.plugin] [10253] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 10254. 2023-02-19 00:57:26 [6ms] [Info] [carb.launcher.plugin] [10253] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 10254. 2023-02-19 00:57:26 [6ms] [Info] [carb.launcher.plugin] [10227] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 10254 from the parent 10227. 2023-02-19 00:57:26 [6ms] [Info] [carb.launcher.plugin] [10227] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 10254 {process = 0x16c13b0} 2023-02-19 00:57:26 [6ms] [Info] [carb.launcher.plugin] [10258] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 10254. 2023-02-19 00:57:26 [6ms] [Info] [carb.launcher.plugin] [10257] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 10254. 2023-02-19 00:57:26 [6ms] [Info] [omni.core] [10227] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 00:57:26 [6ms] [Info] [omni.core] [10227] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 00:57:26 [0ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 00:57:26 [0ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 00:57:26 [1ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10262] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 00:57:26 [2ms] [Info] [omni.server] [10262] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 00:57:26 [2ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 00:57:26 [2ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 00:57:26 [3ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 00:57:26 [3ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 00:57:26 [3ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x10213b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 00:57:26 [3ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 10262. 2023-02-19 00:57:26 [3ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 00:57:26 [3ms] [Info] [carb] [10262] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 00:57:26 [4ms] [Info] [carb.launcher.plugin] [10274] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 10275 from the parent 10262. 2023-02-19 00:57:26 [4ms] [Info] [carb.launcher.plugin] [10274] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 10275. 2023-02-19 00:57:26 [5ms] [Info] [carb.launcher.plugin] [10274] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 10275. 2023-02-19 00:57:26 [5ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 10275 from the parent 10262. 2023-02-19 00:57:26 [5ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 10275 {process = 0x10213b0} 2023-02-19 00:57:26 [5ms] [Info] [carb.launcher.plugin] [10276] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 10275. 2023-02-19 00:57:26 [5ms] [Info] [omni.core] [10262] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 00:57:26 [5ms] [Info] [omni.core] [10262] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 00:57:26 [5ms] [Info] [carb.launcher.plugin] [10277] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 10275. 2023-02-19 00:57:27 [282ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 00:57:27 [286ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 17:57:27,261 - system_log - INFO - reconnecting ... 2023-02-19 00:57:27 [294ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 17:57:27,270 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 00:57:28 [1,296ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 17:57:28,271 - system_log - INFO - reconnecting ... 2023-02-19 00:57:28 [1,298ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 17:57:28,274 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 00:57:28 [2,006ms] [Info] [omni.core] [10266] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 00:57:28 [2,006ms] [Info] [omni.core] [10266] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 00:57:28 [2,006ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 00:57:28 [2,008ms] [Info] [omni.core] [10266] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 00:57:28 [2,009ms] [Info] [omni.core] [10262] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 00:57:28 [2,009ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 00:57:29 [2,039ms] [Warning] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 00:57:29 [2,039ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 00:57:29 [2,054ms] [Warning] [carb] [10262] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 00:57:29 [2,054ms] [Info] [carb] [10262] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 00:57:29 [2,054ms] [Info] [omni.meta-upgrade] [10262] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:getCurrentDbVersion():574: Database version record not found, key: service://metaDatabaseVersion 2023-02-19 00:57:29 [2,054ms] [Info] [omni.meta-upgrade] [10262] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2882: Fresh database, nothing to upgrade 2023-02-19 00:57:29 [2,054ms] [Info] [omni.meta-upgrade] [10262] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:setVersion():552: Meta database upgraded 2023-02-19 00:57:29 [2,055ms] [Info] [omni.core] [10262] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-02-19 00:57:29 [2,057ms] [Info] [omni.core] [10262] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-02-19 00:57:29 [2,062ms] [Info] [omni.core] [10262] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-02-19 00:57:29 [2,062ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-02-19 00:57:29 [2,062ms] [Info] [omni.core] [10267] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-02-19 00:57:29 [2,063ms] [Info] [omni.core] [10269] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-02-19 00:57:29 [2,063ms] [Info] [carb.launcher.plugin] [10269] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7ff7d0001020 for './omni.delta-applier.app'. 2023-02-19 00:57:29 [2,063ms] [Info] [carb.launcher.plugin] [10269] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 10262. 2023-02-19 00:57:29 [2,064ms] [Info] [carb.launcher.plugin] [10274] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 10322 from the parent 10262. 2023-02-19 00:57:29 [2,064ms] [Info] [carb.launcher.plugin] [10274] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 10322. 2023-02-19 00:57:29 [2,066ms] [Info] [carb.launcher.plugin] [10274] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 10322. 2023-02-19 00:57:29 [2,066ms] [Info] [carb.launcher.plugin] [10269] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 10322 from the parent 10262. 2023-02-19 00:57:29 [2,066ms] [Info] [carb.launcher.plugin] [10269] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 10322 {process = 0x7ff7d0001020} 2023-02-19 00:57:29 [2,066ms] [Info] [carb.launcher.plugin] [10324] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 10322. 2023-02-19 00:57:29 [2,066ms] [Info] [carb.launcher.plugin] [10323] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 10322. 2023-02-19 00:57:29 [2,270ms] [Info] [omni.core] [10266] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-02-19 00:57:29 [2,300ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 17:57:29,276 - system_log - INFO - reconnecting ... 2023-02-19 00:57:29 [2,302ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 17:57:29,278 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 00:57:30 [3,303ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 17:57:30,279 - system_log - INFO - reconnecting ... 2023-02-19 00:57:30 [3,305ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 17:57:30,280 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 00:57:31 [4,062ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-02-19 00:57:31 [4,068ms] [Error] [carb] [10262] ../../../source/framework/FileSystem.cpp:move():964: File move failed: '/home/elena/.nvidia-omniverse/logs/Nucleus/Access.log' -> '/home/elena/.nvidia-omniverse/logs/Nucleus/Access_OLD.log' (No such file or directory) 2023-02-19 00:57:31 [4,068ms] [Error] [omni.server] [10262] ../../../source/plugins/omni.server/apilogging/deprecated/ApiLogging.cpp:resetApiLogFile():210: Failed to rename access log file '/home/elena/.nvidia-omniverse/logs/Nucleus/Access.log' to '/home/elena/.nvidia-omniverse/logs/Nucleus/Access_OLD.log' 2023-02-19 00:57:31 [4,068ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-02-19 00:57:31 [4,068ms] [Info] [omni.signalhandler] [10262] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-02-19 00:57:31 [4,068ms] [Info] [omni.signalhandler] [10262] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-02-19 00:57:31 [4,068ms] [Info] [omni.signalhandler] [10262] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-02-19 00:57:31 [4,307ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 17:57:31,282 - system_log - INFO - reconnecting ... 2023-02-19 00:57:31 [4,313ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 17:57:31,288 - system_log - INFO - connected 2023-02-19 00:57:31 [4,314ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-02-19 02:59:34 [0ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 02:59:34 [0ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 02:59:34 [0ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 02:59:34 [1ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 02:59:34 [2ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 02:59:34 [2ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 02:59:34 [2ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 02:59:34 [2ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 02:59:34 [3ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 02:59:34 [3ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 02:59:34 [3ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 02:59:34 [3ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 02:59:34 [3ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 02:59:34 [4ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 02:59:34 [4ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 02:59:34 [4ms] [Info] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 02:59:34 [4ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 02:59:34 [4ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 02:59:34 [4ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 02:59:34 [4ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 02:59:34 [4ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 02:59:34 [5ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 02:59:34 [5ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 02:59:34 [5ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 02:59:34 [5ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 02:59:34 [5ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 02:59:34 [5ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 02:59:34 [5ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 02:59:34 [5ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 02:59:34 [5ms] [Info] [omni.server] [24968] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 02:59:34 [5ms] [Info] [omni.server] [24968] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 02:59:34 [5ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 02:59:34 [5ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 02:59:34 [6ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 02:59:34 [6ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 02:59:34 [6ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x24031c0 for './omni-lft-service/omni-lft-service'. 2023-02-19 02:59:34 [6ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 24968. 2023-02-19 02:59:34 [6ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 02:59:34 [6ms] [Info] [carb] [24968] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 02:59:34 [6ms] [Info] [carb.launcher.plugin] [24985] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 24986 from the parent 24968. 2023-02-19 02:59:34 [6ms] [Info] [carb.launcher.plugin] [24985] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 24986. 2023-02-19 02:59:34 [7ms] [Info] [carb.launcher.plugin] [24985] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 24986. 2023-02-19 02:59:34 [7ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 24986 from the parent 24968. 2023-02-19 02:59:34 [7ms] [Info] [carb.launcher.plugin] [24987] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 24986. 2023-02-19 02:59:34 [7ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 24986 {process = 0x24031c0} 2023-02-19 02:59:34 [7ms] [Info] [omni.core] [24968] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 02:59:34 [7ms] [Info] [carb.launcher.plugin] [24988] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 24986. 2023-02-19 02:59:34 [7ms] [Info] [omni.core] [24968] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 02:59:34 [464ms] [Error] [omni.core] [24988] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 02:59:34 [469ms] [Error] [omni.core] [24988] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 19:59:34,588 - system_log - INFO - reconnecting ... 2023-02-19 02:59:34 [480ms] [Error] [omni.core] [24988] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 19:59:34,599 - system_log - INFO - connected 2023-02-19 02:59:34 [480ms] [Error] [omni.core] [24988] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [24986] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 02:59:34 [520ms] [Info] [carb.launcher.plugin] [24988] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 02:59:34 [520ms] [Info] [carb.launcher.plugin] [24988] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 24986, bytes = 0} 2023-02-19 02:59:34 [520ms] [Info] [carb.launcher.plugin] [24988] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 24986. 2023-02-19 02:59:34 [520ms] [Info] [carb.launcher.plugin] [24987] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 02:59:34 [520ms] [Info] [carb.launcher.plugin] [24987] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 24986, bytes = 0} 2023-02-19 02:59:34 [520ms] [Info] [carb.launcher.plugin] [24987] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 24986. 2023-02-19 02:59:34 [7,327,697ms] [Info] [omni.server.app] [10262] ../../../source/apps/omni.server/main.cpp:main():106: Got signal to exit the application. 2023-02-19 02:59:34 [7,327,697ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():391: OmniverseServer shutdown process starting. 2023-02-19 02:59:34 [7,327,697ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():392: Waiting for any outstanding transactions to complete. 2023-02-19 02:59:34 [7,327,697ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():396: Releasing WebSocket listen ports 2023-02-19 02:59:34 [7,327,697ms] [Info] [omni.server] [25038] ../../../source/plugins/omni.server/ApiConnection.cpp:shutdownCore():70: Shutdown Core 2023-02-19 02:59:34 [7,327,697ms] [Info] [omni.core] [10313] ../../../source/plugins/omni.core/impl/Scheduler.cpp:scheduler():122: Scheduling task is shutted down... 2023-02-19 02:59:34 [7,327,697ms] [Info] [omni.core] [10266] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():422: Finished content remove task 2023-02-19 02:59:34 [7,327,697ms] [Error] [omni.core] [10277] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Shutting down server...wait 2023-02-19 02:59:34 [7,327,697ms] [Warning] [carb] [25038] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 02:59:34 [7,327,698ms] [Info] [omni.core] [25038] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:operator()():849: Content resolver statistics: Total: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0|Temporal: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0|Persistent: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0 2023-02-19 02:59:34 [7,327,698ms] [Info] [omni.server.app] [10262] ../../../source/apps/omni.server/main.cpp:main():120: Releasing the OmniverseServer instance. 2023-02-19 02:59:34 [7,327,698ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():310: OmniverseServerImpl destructor. 2023-02-19 02:59:34 [7,327,698ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():312: Releasing ClientConnectionManager 2023-02-19 02:59:34 [7,327,698ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():315: Releasing ServiceConnectionManager 2023-02-19 02:59:34 [7,327,698ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():318: Releasing LftConnectionManager 2023-02-19 02:59:34 [7,327,698ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():321: Releasing ConnectionManager 2023-02-19 02:59:34 [7,327,699ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 10322 {exitCode = 137, flags = 0x00000002, timeout = 5000} 2023-02-19 02:59:34 [7,327,700ms] [Info] [carb.launcher.plugin] [10323] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 02:59:34 [7,327,700ms] [Info] [carb.launcher.plugin] [10323] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 10322, bytes = 0} 2023-02-19 02:59:34 [7,327,700ms] [Info] [carb.launcher.plugin] [10324] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 02:59:34 [7,327,700ms] [Info] [carb.launcher.plugin] [10323] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 10322. 2023-02-19 02:59:34 [7,327,700ms] [Info] [carb.launcher.plugin] [10324] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 10322, bytes = 0} 2023-02-19 02:59:34 [7,327,700ms] [Info] [carb.launcher.plugin] [10324] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 10322. 2023-02-19 02:59:34 [7,327,701ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x7ff7d0001020 (10322) to exit {timeout = 18446744073709551615} 2023-02-19 02:59:34 [7,327,701ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 02:59:34 [7,327,701ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x7ff7d0001020 {pid = 10322} 2023-02-19 02:59:34 [7,327,701ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x7ff7d0001020 {pid = 10322} 2023-02-19 02:59:34 [7,327,701ms] [Info] [omni.core] [10267] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():457: discovery client: exiting 2023-02-19 02:59:34 [7,327,701ms] [Info] [omni.core] [10262] ../../../source/plugins/common/ChildProcess.cpp:kill():125: Killing child server process ./omni-lft-service/omni-lft-service 2023-02-19 02:59:34 [7,327,701ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 10275 {exitCode = 137, flags = 0x00000003, timeout = 5000} 2023-02-19 02:59:34 [7,327,704ms] [Info] [carb.launcher.plugin] [10277] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 02:59:34 [7,327,704ms] [Info] [carb.launcher.plugin] [10277] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 10275, bytes = 0} 2023-02-19 02:59:34 [7,327,704ms] [Info] [carb.launcher.plugin] [10277] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 10275. 2023-02-19 02:59:34 [7,327,704ms] [Info] [carb.launcher.plugin] [10276] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 02:59:34 [7,327,704ms] [Info] [carb.launcher.plugin] [10276] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 10275, bytes = 0} 2023-02-19 02:59:34 [7,327,704ms] [Info] [carb.launcher.plugin] [10276] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 10275. 2023-02-19 02:59:34 [7,327,705ms] [Info] [omni.core] [10262] ../../../source/plugins/common/ChildProcess.cpp:kill():127: Waiting for child server process ./omni-lft-service/omni-lft-service to exit 2023-02-19 02:59:34 [7,327,705ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x10213b0 (10275) to exit {timeout = 18446744073709551615} 2023-02-19 02:59:34 [7,327,705ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 02:59:34 [7,327,705ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x10213b0 {pid = 10275} 2023-02-19 02:59:34 [7,327,705ms] [Info] [carb.launcher.plugin] [10262] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x10213b0 {pid = 10275} 2023-02-19 02:59:34 [7,327,705ms] [Info] [omni.server] [10262] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():327: Destructor complete. 2023-02-19 02:59:36 [2,008ms] [Info] [omni.core] [24978] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 02:59:36 [2,008ms] [Info] [omni.core] [24978] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 02:59:36 [2,008ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 02:59:36 [2,008ms] [Info] [omni.core] [24978] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 02:59:36 [2,011ms] [Info] [omni.core] [24968] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 02:59:36 [2,011ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 02:59:37 [3,023ms] [Warning] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 02:59:37 [3,023ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 02:59:37 [3,031ms] [Warning] [carb] [24968] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 02:59:37 [3,031ms] [Info] [carb] [24968] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 6 records 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 6 records 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 6 records 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-02-19 02:59:37 [3,031ms] [Info] [omni.meta-upgrade] [24968] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-02-19 02:59:37 [3,031ms] [Info] [omni.core] [24968] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-02-19 02:59:37 [3,032ms] [Info] [omni.core] [24968] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-02-19 02:59:37 [3,035ms] [Info] [omni.core] [24968] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-02-19 02:59:37 [3,035ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-02-19 02:59:37 [3,035ms] [Info] [omni.core] [24979] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-02-19 02:59:37 [3,035ms] [Info] [omni.core] [24979] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-02-19 02:59:37 [3,035ms] [Info] [carb.launcher.plugin] [24979] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7fb4600034c0 for './omni.delta-applier.app'. 2023-02-19 02:59:37 [3,035ms] [Info] [carb.launcher.plugin] [24979] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 24968. 2023-02-19 02:59:37 [3,036ms] [Info] [carb.launcher.plugin] [24985] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 25231 from the parent 24968. 2023-02-19 02:59:37 [3,036ms] [Info] [carb.launcher.plugin] [24985] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 25231. 2023-02-19 02:59:37 [3,037ms] [Info] [carb.launcher.plugin] [24985] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 25231. 2023-02-19 02:59:37 [3,037ms] [Info] [carb.launcher.plugin] [24979] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 25231 from the parent 24968. 2023-02-19 02:59:37 [3,037ms] [Info] [carb.launcher.plugin] [24979] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 25231 {process = 0x7fb4600034c0} 2023-02-19 02:59:37 [3,037ms] [Info] [carb.launcher.plugin] [25233] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 25231. 2023-02-19 02:59:37 [3,038ms] [Info] [carb.launcher.plugin] [25232] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 25231. 2023-02-19 02:59:37 [3,239ms] [Info] [omni.core] [24977] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-02-19 02:59:39 [5,035ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-02-19 02:59:39 [5,040ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-02-19 02:59:39 [5,040ms] [Info] [omni.signalhandler] [24968] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-02-19 02:59:39 [5,040ms] [Info] [omni.signalhandler] [24968] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-02-19 02:59:39 [5,040ms] [Info] [omni.signalhandler] [24968] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-02-19 02:59:46 [12,008ms] [Info] [omni.core] [24976] ../../../source/plugins/common/ChildProcess.cpp:kill():125: Killing child server process ./omni-lft-service/omni-lft-service 2023-02-19 02:59:46 [12,008ms] [Info] [carb.launcher.plugin] [24976] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 24986 {exitCode = 137, flags = 0x00000003, timeout = 5000} 2023-02-19 02:59:46 [12,009ms] [Info] [omni.core] [24976] ../../../source/plugins/common/ChildProcess.cpp:kill():127: Waiting for child server process ./omni-lft-service/omni-lft-service to exit 2023-02-19 02:59:46 [12,009ms] [Info] [carb.launcher.plugin] [24976] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x24031c0 (24986) to exit {timeout = 18446744073709551615} 2023-02-19 02:59:46 [12,009ms] [Info] [carb.launcher.plugin] [24976] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 02:59:46 [12,009ms] [Info] [carb.launcher.plugin] [24976] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x24031c0 {pid = 24986} 2023-02-19 02:59:46 [12,009ms] [Info] [carb.launcher.plugin] [24976] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x24031c0 {pid = 24986} 2023-02-19 02:59:46 [12,009ms] [Info] [carb.launcher.plugin] [24976] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x24031c0 for './omni-lft-service/omni-lft-service'. 2023-02-19 02:59:46 [12,009ms] [Info] [carb.launcher.plugin] [24976] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 24968. 2023-02-19 02:59:46 [12,010ms] [Info] [carb.launcher.plugin] [24985] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 25333 from the parent 24968. 2023-02-19 02:59:46 [12,010ms] [Info] [carb.launcher.plugin] [24985] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 25333. 2023-02-19 02:59:46 [12,011ms] [Info] [carb.launcher.plugin] [24985] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 25333. 2023-02-19 02:59:46 [12,011ms] [Info] [carb.launcher.plugin] [24976] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 25333 from the parent 24968. 2023-02-19 02:59:46 [12,011ms] [Info] [carb.launcher.plugin] [24976] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 25333 {process = 0x24031c0} 2023-02-19 02:59:46 [12,011ms] [Info] [omni.core] [24976] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 02:59:46 [12,011ms] [Info] [omni.core] [24976] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 02:59:46 [12,011ms] [Info] [carb.launcher.plugin] [25334] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 25333. 2023-02-19 02:59:46 [12,011ms] [Info] [carb.launcher.plugin] [25335] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 25333. 2023-02-19 02:59:46 [12,256ms] [Error] [omni.core] [25335] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 02:59:46 [12,259ms] [Error] [omni.core] [25335] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 19:59:46,379 - system_log - INFO - reconnecting ... 2023-02-19 02:59:46 [12,269ms] [Error] [omni.core] [25335] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 19:59:46,388 - system_log - INFO - connected 2023-02-19 02:59:46 [12,269ms] [Error] [omni.core] [25335] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-02-19 03:00:12 [0ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:00:12 [0ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:00:12 [0ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:00:12 [0ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:00:12 [0ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:00:12 [0ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:00:12 [0ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:00:12 [1ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:00:12 [2ms] [Info] [omni.server] [25506] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:00:12 [2ms] [Info] [omni.server] [25506] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:00:12 [2ms] [Info] [omni.server] [25506] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:00:12 [2ms] [Info] [omni.server] [25506] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:00:12 [2ms] [Info] [omni.server] [25506] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:00:12 [2ms] [Info] [omni.server] [25506] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:00:12 [2ms] [Info] [omni.server] [25506] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:00:12 [2ms] [Info] [omni.server] [25506] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:00:12 [2ms] [Info] [omni.server] [25506] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:00:12 [2ms] [Info] [omni.server] [25506] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 03:00:12 [2ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:00:12 [2ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:00:12 [3ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:00:12 [3ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:00:12 [3ms] [Info] [carb.launcher.plugin] [25506] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x10e63b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:00:12 [3ms] [Info] [carb.launcher.plugin] [25506] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 25506. 2023-02-19 03:00:12 [3ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:00:12 [3ms] [Info] [carb] [25506] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:00:12 [3ms] [Info] [carb.launcher.plugin] [25518] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 25519 from the parent 25506. 2023-02-19 03:00:12 [4ms] [Info] [carb.launcher.plugin] [25518] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 25519. 2023-02-19 03:00:12 [4ms] [Info] [carb.launcher.plugin] [25518] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 25519. 2023-02-19 03:00:12 [4ms] [Info] [carb.launcher.plugin] [25506] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 25519 from the parent 25506. 2023-02-19 03:00:12 [4ms] [Info] [carb.launcher.plugin] [25506] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 25519 {process = 0x10e63b0} 2023-02-19 03:00:12 [4ms] [Info] [omni.core] [25506] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:00:12 [4ms] [Info] [omni.core] [25506] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:00:12 [4ms] [Info] [carb.launcher.plugin] [25521] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 25519. 2023-02-19 03:00:12 [4ms] [Info] [carb.launcher.plugin] [25520] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 25519. 2023-02-19 03:00:13 [254ms] [Error] [omni.core] [25521] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:00:13 [258ms] [Error] [omni.core] [25521] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:00:13,084 - system_log - INFO - reconnecting ... 2023-02-19 03:00:13 [267ms] [Error] [omni.core] [25521] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:00:13,094 - system_log - INFO - connected 2023-02-19 03:00:13 [268ms] [Error] [omni.core] [25521] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [25519] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 03:00:13 [295ms] [Info] [carb.launcher.plugin] [25520] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:00:13 [295ms] [Info] [carb.launcher.plugin] [25521] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:00:13 [295ms] [Info] [carb.launcher.plugin] [25520] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 25519, bytes = 0} 2023-02-19 03:00:13 [295ms] [Info] [carb.launcher.plugin] [25521] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 25519, bytes = 0} 2023-02-19 03:00:13 [295ms] [Info] [carb.launcher.plugin] [25520] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 25519. 2023-02-19 03:00:13 [295ms] [Info] [carb.launcher.plugin] [25521] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 25519. 2023-02-19 03:00:14 [2,005ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:00:14 [2,005ms] [Info] [omni.core] [25511] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 03:00:14 [2,005ms] [Info] [omni.core] [25511] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:00:14 [2,005ms] [Info] [omni.core] [25506] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 03:00:14 [2,005ms] [Info] [carb] [25506] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:00:14 [2,005ms] [Warning] [omni.keyvaluedatabase.leveldb] [25506] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 03:00:14 [2,006ms] [Info] [omni.core] [25511] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:00:14 [2,012ms] [Warning] [omni.keyvaluedatabase.leveldb] [25506] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 03:00:14 [2,012ms] [Fatal] [omni.core] [25506] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-02-19 03:00:56 [0ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:00:56 [0ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:00:56 [0ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:00:56 [0ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:00:56 [1ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:00:56 [2ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:00:56 [2ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:00:56 [2ms] [Info] [omni.server] [25757] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:00:56 [2ms] [Info] [omni.server] [25757] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:00:56 [2ms] [Info] [omni.server] [25757] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:00:56 [2ms] [Info] [omni.server] [25757] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:00:56 [2ms] [Info] [omni.server] [25757] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:00:56 [2ms] [Info] [omni.server] [25757] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:00:56 [2ms] [Info] [omni.server] [25757] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:00:56 [2ms] [Info] [omni.server] [25757] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:00:56 [2ms] [Info] [omni.server] [25757] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:00:56 [2ms] [Info] [omni.server] [25757] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 03:00:56 [2ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:00:56 [3ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:00:56 [3ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:00:56 [3ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:00:56 [3ms] [Info] [carb.launcher.plugin] [25757] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x18533b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:00:56 [3ms] [Info] [carb.launcher.plugin] [25757] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 25757. 2023-02-19 03:00:56 [3ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:00:56 [3ms] [Info] [carb] [25757] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:00:56 [4ms] [Info] [carb.launcher.plugin] [25769] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 25770 from the parent 25757. 2023-02-19 03:00:56 [4ms] [Info] [carb.launcher.plugin] [25769] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 25770. 2023-02-19 03:00:56 [5ms] [Info] [carb.launcher.plugin] [25769] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 25770. 2023-02-19 03:00:56 [5ms] [Info] [carb.launcher.plugin] [25757] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 25770 from the parent 25757. 2023-02-19 03:00:56 [5ms] [Info] [carb.launcher.plugin] [25757] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 25770 {process = 0x18533b0} 2023-02-19 03:00:56 [5ms] [Info] [omni.core] [25757] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:00:56 [5ms] [Info] [omni.core] [25757] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:00:56 [5ms] [Info] [carb.launcher.plugin] [25772] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 25770. 2023-02-19 03:00:56 [5ms] [Info] [carb.launcher.plugin] [25773] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 25770. 2023-02-19 03:00:56 [291ms] [Error] [omni.core] [25773] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:00:56 [295ms] [Error] [omni.core] [25773] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:00:56,727 - system_log - INFO - reconnecting ... 2023-02-19 03:00:56 [305ms] [Error] [omni.core] [25773] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:00:56,737 - system_log - INFO - connected 2023-02-19 03:00:56 [305ms] [Error] [omni.core] [25773] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [25770] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 03:00:56 [336ms] [Info] [carb.launcher.plugin] [25772] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:00:56 [336ms] [Info] [carb.launcher.plugin] [25772] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 25770, bytes = 0} 2023-02-19 03:00:56 [336ms] [Info] [carb.launcher.plugin] [25772] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 25770. 2023-02-19 03:00:56 [336ms] [Info] [carb.launcher.plugin] [25773] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:00:56 [336ms] [Info] [carb.launcher.plugin] [25773] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 25770, bytes = 0} 2023-02-19 03:00:56 [336ms] [Info] [carb.launcher.plugin] [25773] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 25770. 2023-02-19 03:00:58 [2,005ms] [Info] [omni.core] [25764] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 03:00:58 [2,005ms] [Info] [omni.core] [25764] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:00:58 [2,005ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:00:58 [2,006ms] [Info] [omni.core] [25764] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:00:58 [2,006ms] [Info] [omni.core] [25757] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 03:00:58 [2,006ms] [Info] [carb] [25757] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:00:58 [2,006ms] [Warning] [omni.keyvaluedatabase.leveldb] [25757] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 03:00:58 [2,011ms] [Warning] [omni.keyvaluedatabase.leveldb] [25757] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 03:00:58 [2,011ms] [Fatal] [omni.core] [25757] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-02-19 03:03:22 [0ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:03:22 [0ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:03:22 [0ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:03:22 [1ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:03:22 [2ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:03:22 [2ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:03:22 [2ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:03:22 [2ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:03:22 [3ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:03:22 [3ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:03:22 [3ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:03:22 [3ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:03:22 [4ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:03:22 [4ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:03:22 [4ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:03:22 [5ms] [Info] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:03:22 [5ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:03:22 [5ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:03:22 [5ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:03:22 [5ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:03:22 [5ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:03:22 [6ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:03:22 [6ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:03:22 [6ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:03:22 [6ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:03:22 [6ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:03:22 [6ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:03:22 [6ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:03:22 [6ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:03:22 [7ms] [Info] [omni.server] [27041] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:03:22 [7ms] [Info] [omni.server] [27041] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 03:03:22 [8ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:03:22 [8ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:03:22 [9ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:03:22 [11ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:03:22 [11ms] [Info] [carb.launcher.plugin] [27041] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1ca11c0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:03:22 [11ms] [Info] [carb.launcher.plugin] [27041] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 27041. 2023-02-19 03:03:22 [11ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:03:22 [11ms] [Info] [carb] [27041] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:03:22 [12ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 27065 from the parent 27041. 2023-02-19 03:03:22 [12ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 27065. 2023-02-19 03:03:22 [13ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 27065. 2023-02-19 03:03:22 [13ms] [Info] [carb.launcher.plugin] [27041] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 27065 from the parent 27041. 2023-02-19 03:03:22 [13ms] [Info] [carb.launcher.plugin] [27067] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 27065. 2023-02-19 03:03:22 [13ms] [Info] [carb.launcher.plugin] [27041] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 27065 {process = 0x1ca11c0} 2023-02-19 03:03:22 [13ms] [Info] [omni.core] [27041] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:03:22 [13ms] [Info] [carb.launcher.plugin] [27068] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 27065. 2023-02-19 03:03:22 [13ms] [Info] [omni.core] [27041] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:03:23 [342ms] [Error] [omni.core] [27068] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:03:23 [346ms] [Error] [omni.core] [27068] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:03:23,183 - system_log - INFO - reconnecting ... 2023-02-19 03:03:23 [355ms] [Error] [omni.core] [27068] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:03:23,193 - system_log - INFO - connected 2023-02-19 03:03:23 [356ms] [Error] [omni.core] [27068] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [27065] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 03:03:23 [390ms] [Info] [carb.launcher.plugin] [27068] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:03:23 [390ms] [Info] [carb.launcher.plugin] [27068] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 27065, bytes = 0} 2023-02-19 03:03:23 [390ms] [Info] [carb.launcher.plugin] [27068] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 27065. 2023-02-19 03:03:23 [390ms] [Info] [carb.launcher.plugin] [27067] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:03:23 [390ms] [Info] [carb.launcher.plugin] [27067] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 27065, bytes = 0} 2023-02-19 03:03:23 [390ms] [Info] [carb.launcher.plugin] [27067] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 27065. 2023-02-19 03:03:23 [229,347ms] [Info] [omni.server.app] [24968] ../../../source/apps/omni.server/main.cpp:main():106: Got signal to exit the application. 2023-02-19 03:03:23 [229,347ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():391: OmniverseServer shutdown process starting. 2023-02-19 03:03:23 [229,347ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():392: Waiting for any outstanding transactions to complete. 2023-02-19 03:03:23 [229,347ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():396: Releasing WebSocket listen ports 2023-02-19 03:03:23 [229,347ms] [Info] [omni.server] [27176] ../../../source/plugins/omni.server/ApiConnection.cpp:shutdownCore():70: Shutdown Core 2023-02-19 03:03:23 [229,347ms] [Info] [omni.core] [25219] ../../../source/plugins/omni.core/impl/Scheduler.cpp:scheduler():122: Scheduling task is shutted down... 2023-02-19 03:03:23 [229,348ms] [Info] [omni.core] [24976] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():422: Finished content remove task 2023-02-19 03:03:23 [229,348ms] [Error] [omni.core] [25335] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Shutting down server...wait 2023-02-19 03:03:23 [229,348ms] [Warning] [carb] [27176] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 03:03:23 [229,348ms] [Info] [omni.core] [27176] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:operator()():849: Content resolver statistics: Total: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0|Temporal: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0|Persistent: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0 2023-02-19 03:03:23 [229,349ms] [Info] [omni.server.app] [24968] ../../../source/apps/omni.server/main.cpp:main():120: Releasing the OmniverseServer instance. 2023-02-19 03:03:23 [229,349ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():310: OmniverseServerImpl destructor. 2023-02-19 03:03:23 [229,349ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():312: Releasing ClientConnectionManager 2023-02-19 03:03:23 [229,349ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():315: Releasing ServiceConnectionManager 2023-02-19 03:03:23 [229,349ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():318: Releasing LftConnectionManager 2023-02-19 03:03:23 [229,349ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():321: Releasing ConnectionManager 2023-02-19 03:03:23 [229,349ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 25231 {exitCode = 137, flags = 0x00000002, timeout = 5000} 2023-02-19 03:03:23 [229,350ms] [Info] [carb.launcher.plugin] [25233] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:03:23 [229,350ms] [Info] [carb.launcher.plugin] [25232] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:03:23 [229,351ms] [Info] [carb.launcher.plugin] [25232] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 25231, bytes = 0} 2023-02-19 03:03:23 [229,351ms] [Info] [carb.launcher.plugin] [25232] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 25231. 2023-02-19 03:03:23 [229,351ms] [Info] [carb.launcher.plugin] [25233] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 25231, bytes = 0} 2023-02-19 03:03:23 [229,351ms] [Info] [carb.launcher.plugin] [25233] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 25231. 2023-02-19 03:03:23 [229,352ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x7fb4600034c0 (25231) to exit {timeout = 18446744073709551615} 2023-02-19 03:03:23 [229,352ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 03:03:23 [229,352ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x7fb4600034c0 {pid = 25231} 2023-02-19 03:03:23 [229,352ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x7fb4600034c0 {pid = 25231} 2023-02-19 03:03:23 [229,352ms] [Info] [omni.core] [24977] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():457: discovery client: exiting 2023-02-19 03:03:23 [229,353ms] [Info] [omni.core] [24968] ../../../source/plugins/common/ChildProcess.cpp:kill():125: Killing child server process ./omni-lft-service/omni-lft-service 2023-02-19 03:03:23 [229,353ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 25333 {exitCode = 137, flags = 0x00000003, timeout = 5000} 2023-02-19 03:03:23 [229,355ms] [Info] [carb.launcher.plugin] [25334] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:03:23 [229,355ms] [Info] [carb.launcher.plugin] [25334] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 25333, bytes = 0} 2023-02-19 03:03:23 [229,355ms] [Info] [carb.launcher.plugin] [25334] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 25333. 2023-02-19 03:03:23 [229,355ms] [Info] [carb.launcher.plugin] [25335] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:03:23 [229,355ms] [Info] [carb.launcher.plugin] [25335] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 25333, bytes = 0} 2023-02-19 03:03:23 [229,355ms] [Info] [carb.launcher.plugin] [25335] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 25333. 2023-02-19 03:03:23 [229,357ms] [Info] [omni.core] [24968] ../../../source/plugins/common/ChildProcess.cpp:kill():127: Waiting for child server process ./omni-lft-service/omni-lft-service to exit 2023-02-19 03:03:23 [229,357ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x24031c0 (25333) to exit {timeout = 18446744073709551615} 2023-02-19 03:03:23 [229,357ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 03:03:23 [229,357ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x24031c0 {pid = 25333} 2023-02-19 03:03:23 [229,357ms] [Info] [carb.launcher.plugin] [24968] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x24031c0 {pid = 25333} 2023-02-19 03:03:23 [229,357ms] [Info] [omni.server] [24968] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():327: Destructor complete. 2023-02-19 03:03:24 [2,014ms] [Info] [omni.core] [27056] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 03:03:24 [2,014ms] [Info] [omni.core] [27056] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:03:24 [2,014ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:03:24 [2,015ms] [Info] [omni.core] [27056] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:03:24 [2,015ms] [Info] [omni.core] [27041] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 03:03:24 [2,016ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:03:25 [2,487ms] [Warning] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 03:03:25 [2,487ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:03:25 [2,517ms] [Warning] [carb] [27041] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 03:03:25 [2,517ms] [Info] [carb] [27041] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 7 records 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 7 records 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 7 records 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-02-19 03:03:25 [2,517ms] [Info] [omni.meta-upgrade] [27041] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-02-19 03:03:25 [2,518ms] [Info] [omni.core] [27041] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-02-19 03:03:25 [2,519ms] [Info] [omni.core] [27041] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-02-19 03:03:25 [2,522ms] [Info] [omni.core] [27041] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-02-19 03:03:25 [2,522ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-02-19 03:03:25 [2,522ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-02-19 03:03:25 [2,523ms] [Info] [omni.core] [27056] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-02-19 03:03:25 [2,523ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7f52e40064a0 for './omni.delta-applier.app'. 2023-02-19 03:03:25 [2,523ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 27041. 2023-02-19 03:03:25 [2,524ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 27302 from the parent 27041. 2023-02-19 03:03:25 [2,524ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 27302. 2023-02-19 03:03:25 [2,525ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 27302. 2023-02-19 03:03:25 [2,525ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 27302 from the parent 27041. 2023-02-19 03:03:25 [2,525ms] [Info] [carb.launcher.plugin] [27309] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 27302. 2023-02-19 03:03:25 [2,525ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 27302 {process = 0x7f52e40064a0} 2023-02-19 03:03:25 [2,525ms] [Info] [carb.launcher.plugin] [27310] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 27302. 2023-02-19 03:03:25 [2,727ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-02-19 03:03:27 [4,522ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-02-19 03:03:27 [4,527ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-02-19 03:03:27 [4,527ms] [Info] [omni.signalhandler] [27041] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-02-19 03:03:27 [4,527ms] [Info] [omni.signalhandler] [27041] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-02-19 03:03:27 [4,527ms] [Info] [omni.signalhandler] [27041] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-02-19 03:03:34 [12,014ms] [Info] [omni.core] [27057] ../../../source/plugins/common/ChildProcess.cpp:kill():125: Killing child server process ./omni-lft-service/omni-lft-service 2023-02-19 03:03:34 [12,014ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 27065 {exitCode = 137, flags = 0x00000003, timeout = 5000} 2023-02-19 03:03:34 [12,015ms] [Info] [omni.core] [27057] ../../../source/plugins/common/ChildProcess.cpp:kill():127: Waiting for child server process ./omni-lft-service/omni-lft-service to exit 2023-02-19 03:03:34 [12,015ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x1ca11c0 (27065) to exit {timeout = 18446744073709551615} 2023-02-19 03:03:34 [12,015ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 03:03:34 [12,015ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x1ca11c0 {pid = 27065} 2023-02-19 03:03:34 [12,015ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x1ca11c0 {pid = 27065} 2023-02-19 03:03:34 [12,015ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1ca11c0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:03:34 [12,015ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 27041. 2023-02-19 03:03:34 [12,016ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 27517 from the parent 27041. 2023-02-19 03:03:34 [12,016ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 27517. 2023-02-19 03:03:34 [12,018ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 27517. 2023-02-19 03:03:34 [12,018ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 27517 from the parent 27041. 2023-02-19 03:03:34 [12,018ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 27517 {process = 0x1ca11c0} 2023-02-19 03:03:34 [12,018ms] [Info] [omni.core] [27057] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:03:34 [12,018ms] [Info] [omni.core] [27057] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:03:34 [12,018ms] [Info] [carb.launcher.plugin] [27518] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 27517. 2023-02-19 03:03:34 [12,018ms] [Info] [carb.launcher.plugin] [27519] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 27517. 2023-02-19 03:03:35 [12,265ms] [Error] [omni.core] [27519] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:03:35 [12,269ms] [Error] [omni.core] [27519] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:03:35,106 - system_log - INFO - reconnecting ... 2023-02-19 03:03:35 [12,279ms] [Error] [omni.core] [27519] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:03:35,117 - system_log - INFO - connected 2023-02-19 03:03:35 [12,280ms] [Error] [omni.core] [27519] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-02-19 03:05:30 [0ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:05:30 [0ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:05:30 [0ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:05:30 [0ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:05:30 [0ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:05:30 [0ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:05:30 [0ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:05:30 [0ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28854] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:05:30 [1ms] [Info] [omni.server] [28854] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:05:30 [1ms] [Info] [omni.server] [28854] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:05:30 [2ms] [Info] [omni.server] [28854] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:05:30 [2ms] [Info] [omni.server] [28854] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:05:30 [2ms] [Info] [omni.server] [28854] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:05:30 [2ms] [Info] [omni.server] [28854] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:05:30 [2ms] [Info] [omni.server] [28854] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:05:30 [2ms] [Info] [omni.server] [28854] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:05:30 [2ms] [Info] [omni.server] [28854] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:05:30 [2ms] [Info] [omni.server] [28854] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "" } } } 2023-02-19 03:05:30 [2ms] [Info] [carb] [28854] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28854] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28854] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:05:30 [3ms] [Info] [carb] [28854] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:05:30 [3ms] [Info] [carb.launcher.plugin] [28854] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1f463b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:05:30 [3ms] [Info] [carb.launcher.plugin] [28854] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 28854. 2023-02-19 03:05:30 [3ms] [Info] [carb] [28854] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:05:30 [3ms] [Info] [carb] [28854] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:05:30 [3ms] [Info] [carb.launcher.plugin] [28878] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 28879 from the parent 28854. 2023-02-19 03:05:30 [3ms] [Info] [carb.launcher.plugin] [28878] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 28879. 2023-02-19 03:05:30 [4ms] [Info] [carb.launcher.plugin] [28878] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 28879. 2023-02-19 03:05:30 [4ms] [Info] [carb.launcher.plugin] [28854] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 28879 from the parent 28854. 2023-02-19 03:05:30 [4ms] [Info] [carb.launcher.plugin] [28854] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 28879 {process = 0x1f463b0} 2023-02-19 03:05:30 [4ms] [Info] [omni.core] [28854] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:05:30 [4ms] [Info] [omni.core] [28854] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:05:30 [4ms] [Info] [carb.launcher.plugin] [28883] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 28879. 2023-02-19 03:05:30 [4ms] [Info] [carb.launcher.plugin] [28882] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 28879. 2023-02-19 03:05:30 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:05:30 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:05:30 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:05:30 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:05:30 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:05:30 [2ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:05:30 [3ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:05:30 [3ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:05:30 [3ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:05:30 [3ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:05:30 [3ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:05:30 [3ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:05:30 [3ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:05:30 [3ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:05:30 [3ms] [Info] [omni.server] [28889] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:05:30 [4ms] [Info] [omni.server] [28889] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 03:05:30 [4ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:05:30 [4ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:05:30 [5ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:05:30 [5ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:05:30 [5ms] [Info] [carb.launcher.plugin] [28889] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x15d83b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:05:30 [5ms] [Info] [carb.launcher.plugin] [28889] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 28889. 2023-02-19 03:05:30 [5ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:05:30 [5ms] [Info] [carb] [28889] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:05:30 [6ms] [Info] [carb.launcher.plugin] [28901] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 28902 from the parent 28889. 2023-02-19 03:05:30 [6ms] [Info] [carb.launcher.plugin] [28901] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 28902. 2023-02-19 03:05:30 [7ms] [Info] [carb.launcher.plugin] [28901] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 28902. 2023-02-19 03:05:30 [7ms] [Info] [carb.launcher.plugin] [28889] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 28902 from the parent 28889. 2023-02-19 03:05:30 [7ms] [Info] [carb.launcher.plugin] [28889] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 28902 {process = 0x15d83b0} 2023-02-19 03:05:30 [7ms] [Info] [carb.launcher.plugin] [28903] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 28902. 2023-02-19 03:05:30 [7ms] [Info] [omni.core] [28889] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:05:30 [7ms] [Info] [omni.core] [28889] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:05:30 [7ms] [Info] [carb.launcher.plugin] [28904] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 28902. 2023-02-19 03:05:31 [321ms] [Error] [omni.core] [28904] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:05:31 [325ms] [Error] [omni.core] [28904] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:05:31,257 - system_log - INFO - reconnecting ... 2023-02-19 03:05:31 [334ms] [Error] [omni.core] [28904] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:05:31,267 - system_log - INFO - connected 2023-02-19 03:05:31 [335ms] [Error] [omni.core] [28904] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [28902] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 03:05:31 [365ms] [Info] [carb.launcher.plugin] [28903] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:05:31 [365ms] [Info] [carb.launcher.plugin] [28903] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 28902, bytes = 0} 2023-02-19 03:05:31 [365ms] [Info] [carb.launcher.plugin] [28904] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:05:31 [365ms] [Info] [carb.launcher.plugin] [28903] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 28902. 2023-02-19 03:05:31 [365ms] [Info] [carb.launcher.plugin] [28904] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 28902, bytes = 0} 2023-02-19 03:05:31 [365ms] [Info] [carb.launcher.plugin] [28904] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 28902. 2023-02-19 03:05:32 [2,008ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:05:32 [2,008ms] [Info] [omni.core] [28894] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 03:05:32 [2,008ms] [Info] [omni.core] [28894] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:05:32 [2,009ms] [Info] [omni.core] [28894] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:05:32 [2,009ms] [Info] [omni.core] [28889] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 03:05:32 [2,009ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:05:32 [2,009ms] [Warning] [omni.keyvaluedatabase.leveldb] [28889] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 03:05:32 [2,018ms] [Warning] [omni.keyvaluedatabase.leveldb] [28889] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 03:05:32 [2,018ms] [Fatal] [omni.core] [28889] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-02-19 03:06:56 [214,098ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:06:56 [214,101ms] [Info] [omni.server.app] [27041] ../../../source/apps/omni.server/main.cpp:main():106: Got signal to exit the application. 2023-02-19 03:06:56 [214,101ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():391: OmniverseServer shutdown process starting. 2023-02-19 03:06:56 [214,101ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():392: Waiting for any outstanding transactions to complete. 2023-02-19 03:06:56 [214,101ms] [Info] [omni.server] [27041] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():396: Releasing WebSocket listen ports 2023-02-19 03:06:56 [214,101ms] [Info] [omni.server] [30155] ../../../source/plugins/omni.server/ApiConnection.cpp:shutdownCore():70: Shutdown Core 2023-02-19 03:06:56 [214,106ms] [Info] [carb.launcher.plugin] [27309] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:06:56 [214,106ms] [Info] [carb.launcher.plugin] [27310] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:06:56 [214,106ms] [Info] [carb.launcher.plugin] [27309] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 27302, bytes = 0} 2023-02-19 03:06:56 [214,106ms] [Info] [carb.launcher.plugin] [27310] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 27302, bytes = 0} 2023-02-19 03:06:56 [214,106ms] [Info] [carb.launcher.plugin] [27309] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 27302. 2023-02-19 03:06:56 [214,106ms] [Info] [carb.launcher.plugin] [27310] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 27302. 2023-02-19 03:06:56 [214,115ms] [Error] [omni.core] [27519] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Shutting down server...wait 2023-02-19 03:06:57 [214,545ms] [Error] [omni.core] [27519] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server is stopped 2023-02-19 03:06:57 [214,584ms] [Info] [carb.launcher.plugin] [27519] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:06:57 [214,584ms] [Info] [carb.launcher.plugin] [27519] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 27517, bytes = 0} 2023-02-19 03:06:57 [214,584ms] [Info] [carb.launcher.plugin] [27518] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:06:57 [214,584ms] [Info] [carb.launcher.plugin] [27519] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 27517. 2023-02-19 03:06:57 [214,584ms] [Info] [carb.launcher.plugin] [27518] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 27517, bytes = 0} 2023-02-19 03:06:57 [214,584ms] [Info] [carb.launcher.plugin] [27518] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 27517. 2023-02-19 03:06:57 [214,865ms] [Warning] [omni.core] [27057] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():604: Restarting delta applier process because ping timed out. 2023-02-19 03:06:57 [214,865ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 27302 {exitCode = 137, flags = 0x00000002, timeout = 5000} 2023-02-19 03:06:57 [214,867ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x7f52e40064a0 (27302) to exit {timeout = 18446744073709551615} 2023-02-19 03:06:57 [214,867ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 03:06:57 [214,867ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x7f52e40064a0 {pid = 27302} 2023-02-19 03:06:57 [214,867ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x7f52e40064a0 {pid = 27302} 2023-02-19 03:06:57 [214,867ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7f52e40064a0 for './omni.delta-applier.app'. 2023-02-19 03:06:57 [214,867ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 27041. 2023-02-19 03:06:57 [214,868ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 30156 from the parent 27041. 2023-02-19 03:06:57 [214,868ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 30156. 2023-02-19 03:06:57 [214,869ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 30156. 2023-02-19 03:06:57 [214,869ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 30156 from the parent 27041. 2023-02-19 03:06:57 [214,869ms] [Info] [carb.launcher.plugin] [27057] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 30156 {process = 0x7f52e40064a0} 2023-02-19 03:06:57 [214,869ms] [Info] [carb.launcher.plugin] [30157] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 30156. 2023-02-19 03:06:57 [214,869ms] [Info] [carb.launcher.plugin] [30158] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 30156. 2023-02-19 03:06:57 [215,099ms] [Info] [omni.core] [27056] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:06:57 [215,100ms] [Info] [omni.core] [27056] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:06:57 [215,100ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:06:58 [215,472ms] [Info] [omni.core] [27058] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-02-19 03:06:58 [216,100ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:06:58 [216,100ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:06:58 [216,101ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:06:59 [217,101ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:06:59 [217,101ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:06:59 [217,102ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:07:00 [218,102ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:00 [218,102ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:00 [218,102ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:07:01 [219,102ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:01 [219,103ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:01 [219,103ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:07:02 [220,103ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:02 [220,103ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:02 [220,103ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:07:03 [221,104ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:03 [221,104ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:03 [221,104ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:07:04 [222,104ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:04 [222,105ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:04 [222,105ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:07:05 [223,105ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:05 [223,105ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:05 [223,105ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:07:06 [224,022ms] [Info] [omni.core] [27056] ../../../source/plugins/common/ChildProcess.cpp:kill():125: Killing child server process ./omni-lft-service/omni-lft-service 2023-02-19 03:07:06 [224,022ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 27517 {exitCode = 137, flags = 0x00000003, timeout = 5000} 2023-02-19 03:07:06 [224,023ms] [Info] [omni.core] [27056] ../../../source/plugins/common/ChildProcess.cpp:kill():127: Waiting for child server process ./omni-lft-service/omni-lft-service to exit 2023-02-19 03:07:06 [224,023ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x1ca11c0 (27517) to exit {timeout = 18446744073709551615} 2023-02-19 03:07:06 [224,023ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 03:07:06 [224,023ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x1ca11c0 {pid = 27517} 2023-02-19 03:07:06 [224,023ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x1ca11c0 {pid = 27517} 2023-02-19 03:07:06 [224,023ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1ca11c0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:07:06 [224,023ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 27041. 2023-02-19 03:07:06 [224,026ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 30232 from the parent 27041. 2023-02-19 03:07:06 [224,026ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 30232. 2023-02-19 03:07:06 [224,030ms] [Info] [carb.launcher.plugin] [27064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 30232. 2023-02-19 03:07:06 [224,030ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 30232 from the parent 27041. 2023-02-19 03:07:06 [224,030ms] [Info] [carb.launcher.plugin] [27056] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 30232 {process = 0x1ca11c0} 2023-02-19 03:07:06 [224,030ms] [Info] [omni.core] [27056] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:07:06 [224,030ms] [Info] [omni.core] [27056] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:07:06 [224,031ms] [Info] [carb.launcher.plugin] [30233] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 30232. 2023-02-19 03:07:06 [224,031ms] [Info] [carb.launcher.plugin] [30234] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 30232. 2023-02-19 03:07:06 [224,106ms] [Info] [omni.core] [27058] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:06 [224,106ms] [Info] [omni.core] [27058] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:06 [224,106ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:07:07 [224,276ms] [Error] [omni.core] [30234] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:07:07 [224,279ms] [Error] [omni.core] [30234] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:07:07,117 - system_log - INFO - reconnecting ... 2023-02-19 03:07:07 [224,288ms] [Error] [omni.core] [30234] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:07:07,125 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 03:07:07 [225,107ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:07 [225,108ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:07 [225,109ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:07:08 [225,289ms] [Error] [omni.core] [30234] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 20:07:08,127 - system_log - INFO - reconnecting ... 2023-02-19 03:07:08 [225,292ms] [Error] [omni.core] [30234] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:07:08,129 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 03:07:08 [226,109ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:08 [226,111ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:08 [226,111ms] [Info] [omni.core] [27057] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():440: discovery client: failed to issue a call (5 / Not connected) 2023-02-19 03:07:09 [226,293ms] [Error] [omni.core] [30234] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 20:07:09,131 - system_log - INFO - reconnecting ... 2023-02-19 03:07:09 [226,296ms] [Error] [omni.core] [30234] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:07:09,133 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 03:07:09 [227,112ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:09 [227,112ms] [Info] [omni.core] [27055] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:09 [227,113ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:07:10 [227,297ms] [Error] [omni.core] [30234] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 20:07:10,135 - system_log - INFO - reconnecting ... 2023-02-19 03:07:10 [227,300ms] [Error] [omni.core] [30234] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:07:10,137 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 03:07:10 [228,113ms] [Info] [omni.core] [27058] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:07:10 [228,114ms] [Info] [omni.core] [27058] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:07:10 [228,114ms] [Info] [omni.core] [27063] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 03:32:06 [0ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:32:06 [1ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:32:06 [2ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:32:06 [6ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:32:06 [13ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:32:06 [13ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:32:06 [16ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:32:06 [17ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:32:06 [17ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:32:06 [18ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:32:06 [19ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:32:06 [19ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:32:06 [20ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:32:06 [22ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:32:06 [24ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:32:06 [27ms] [Info] [carb] [2846] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:32:06 [27ms] [Info] [carb] [2846] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:32:06 [27ms] [Info] [carb] [2846] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:32:06 [27ms] [Info] [carb] [2846] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:32:06 [27ms] [Info] [carb] [2846] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:32:06 [27ms] [Info] [carb] [2846] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:32:06 [28ms] [Info] [omni.server] [2846] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:32:06 [28ms] [Info] [omni.server] [2846] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:32:06 [28ms] [Info] [omni.server] [2846] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:32:06 [28ms] [Info] [omni.server] [2846] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:32:06 [28ms] [Info] [omni.server] [2846] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:32:06 [28ms] [Info] [omni.server] [2846] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:32:06 [28ms] [Info] [omni.server] [2846] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:32:06 [28ms] [Info] [omni.server] [2846] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:32:06 [28ms] [Info] [omni.server] [2846] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:32:06 [29ms] [Info] [omni.server] [2846] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 03:32:06 [29ms] [Info] [carb] [2846] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:32:06 [30ms] [Info] [carb] [2846] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:32:06 [31ms] [Info] [carb] [2846] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:32:06 [32ms] [Info] [carb] [2846] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:32:06 [33ms] [Info] [carb.launcher.plugin] [2846] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xa751c0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:32:06 [33ms] [Info] [carb.launcher.plugin] [2846] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 2846. 2023-02-19 03:32:06 [33ms] [Info] [carb] [2846] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:32:06 [33ms] [Info] [carb] [2846] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:32:06 [33ms] [Info] [carb.launcher.plugin] [2860] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 2861 from the parent 2846. 2023-02-19 03:32:06 [33ms] [Info] [carb.launcher.plugin] [2860] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 2861. 2023-02-19 03:32:06 [34ms] [Info] [carb.launcher.plugin] [2860] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 2861. 2023-02-19 03:32:06 [34ms] [Info] [carb.launcher.plugin] [2846] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 2861 from the parent 2846. 2023-02-19 03:32:06 [34ms] [Info] [carb.launcher.plugin] [2862] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 2861. 2023-02-19 03:32:06 [34ms] [Info] [carb.launcher.plugin] [2846] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 2861 {process = 0xa751c0} 2023-02-19 03:32:06 [34ms] [Info] [carb.launcher.plugin] [2863] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 2861. 2023-02-19 03:32:06 [34ms] [Info] [omni.core] [2846] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:32:06 [34ms] [Info] [omni.core] [2846] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:32:06 [526ms] [Error] [omni.core] [2863] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:32:06 [532ms] [Error] [omni.core] [2863] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:32:06,933 - system_log - INFO - reconnecting ... 2023-02-19 03:32:06 [540ms] [Error] [omni.core] [2863] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:32:06,942 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:32:08 [0ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:32:08 [1ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:32:08 [1ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:32:08 [1ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:32:08 [1ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:32:08 [1ms] [Info] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:32:08 [1ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:32:08 [1ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:32:08 [1ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:32:08 [1ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:32:08 [1ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:32:08 [1ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:32:08 [1ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:32:08 [1ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:32:08 [1ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:32:08 [1ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:32:08 [1ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:32:08 [1ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:32:08 [1ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:32:08 [1ms] [Info] [omni.server] [3092] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:32:08 [2ms] [Info] [omni.server] [3092] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 03:32:08 [2ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:32:08 [2ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:32:08 [2ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:32:08 [3ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:32:08 [3ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x18fe140 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:32:08 [3ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3092. 2023-02-19 03:32:08 [3ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:32:08 [3ms] [Info] [carb] [3092] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:32:08 [3ms] [Info] [carb.launcher.plugin] [3104] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3105 from the parent 3092. 2023-02-19 03:32:08 [3ms] [Info] [carb.launcher.plugin] [3104] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3105. 2023-02-19 03:32:08 [4ms] [Info] [carb.launcher.plugin] [3104] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3105. 2023-02-19 03:32:08 [4ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3105 from the parent 3092. 2023-02-19 03:32:08 [4ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3105 {process = 0x18fe140} 2023-02-19 03:32:08 [4ms] [Info] [omni.core] [3092] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:32:08 [4ms] [Info] [omni.core] [3092] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:32:08 [4ms] [Info] [carb.launcher.plugin] [3107] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3105. 2023-02-19 03:32:08 [4ms] [Info] [carb.launcher.plugin] [3106] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3105. 2023-02-19 03:32:08 [248ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:32:08 [252ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:32:08,295 - system_log - INFO - reconnecting ... 2023-02-19 03:32:08 [261ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:32:08,304 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 03:32:09 [1,263ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 20:32:09,306 - system_log - INFO - reconnecting ... 2023-02-19 03:32:09 [1,265ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:32:09,308 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 03:32:10 [2,005ms] [Info] [omni.core] [3097] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 03:32:10 [2,006ms] [Info] [omni.core] [3097] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:32:10 [2,007ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:32:10 [2,009ms] [Info] [omni.core] [3097] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:32:10 [2,017ms] [Info] [omni.core] [3092] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 03:32:10 [2,017ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:32:10 [2,041ms] [Warning] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 03:32:10 [2,041ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:32:10 [2,054ms] [Warning] [carb] [3092] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 03:32:10 [2,054ms] [Info] [carb] [3092] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:32:10 [2,054ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-02-19 03:32:10 [2,054ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-02-19 03:32:10 [2,054ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 7 records 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 7 records 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 7 records 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-02-19 03:32:10 [2,055ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-02-19 03:32:10 [2,056ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-02-19 03:32:10 [2,056ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-02-19 03:32:10 [2,056ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-02-19 03:32:10 [2,056ms] [Info] [omni.meta-upgrade] [3092] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-02-19 03:32:10 [2,058ms] [Info] [omni.core] [3092] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-02-19 03:32:10 [2,067ms] [Info] [omni.core] [3092] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-02-19 03:32:10 [2,074ms] [Info] [omni.core] [3092] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-02-19 03:32:10 [2,075ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-02-19 03:32:10 [2,075ms] [Info] [omni.core] [3098] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-02-19 03:32:10 [2,076ms] [Info] [omni.core] [3099] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-02-19 03:32:10 [2,076ms] [Info] [carb.launcher.plugin] [3099] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7f4ec40013b0 for './omni.delta-applier.app'. 2023-02-19 03:32:10 [2,076ms] [Info] [carb.launcher.plugin] [3099] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 3092. 2023-02-19 03:32:10 [2,077ms] [Info] [carb.launcher.plugin] [3104] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3133 from the parent 3092. 2023-02-19 03:32:10 [2,077ms] [Info] [carb.launcher.plugin] [3104] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3133. 2023-02-19 03:32:10 [2,079ms] [Info] [carb.launcher.plugin] [3104] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3133. 2023-02-19 03:32:10 [2,079ms] [Info] [carb.launcher.plugin] [3099] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3133 from the parent 3092. 2023-02-19 03:32:10 [2,079ms] [Info] [carb.launcher.plugin] [3099] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3133 {process = 0x7f4ec40013b0} 2023-02-19 03:32:10 [2,080ms] [Info] [carb.launcher.plugin] [3134] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3133. 2023-02-19 03:32:10 [2,080ms] [Info] [carb.launcher.plugin] [3135] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3133. 2023-02-19 03:32:10 [2,268ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 20:32:10,311 - system_log - INFO - reconnecting ... 2023-02-19 03:32:10 [2,271ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:32:10,313 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 03:32:10 [2,387ms] [Info] [omni.core] [3099] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-02-19 03:32:11 [3,273ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 20:32:11,316 - system_log - INFO - reconnecting ... 2023-02-19 03:32:11 [3,275ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:32:11,318 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 03:32:12 [4,075ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-02-19 03:32:12 [4,083ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-02-19 03:32:12 [4,083ms] [Info] [omni.signalhandler] [3092] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-02-19 03:32:12 [4,083ms] [Info] [omni.signalhandler] [3092] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-02-19 03:32:12 [4,083ms] [Info] [omni.signalhandler] [3092] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-02-19 03:32:12 [4,278ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 20:32:12,321 - system_log - INFO - reconnecting ... 2023-02-19 03:32:12 [4,287ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:32:12,330 - system_log - INFO - connected 2023-02-19 03:32:12 [4,288ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-02-19 03:34:02 [0ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:34:02 [0ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:34:02 [0ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:34:02 [0ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:34:02 [0ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:34:02 [0ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:34:02 [0ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:34:02 [0ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:34:02 [0ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:34:02 [0ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:34:02 [1ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:34:02 [1ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:34:02 [1ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:34:02 [1ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:34:02 [1ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:34:02 [1ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:34:02 [1ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:34:02 [1ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:34:02 [1ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:34:02 [1ms] [Info] [omni.server] [3530] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:34:02 [2ms] [Info] [omni.server] [3530] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 03:34:02 [2ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:34:02 [2ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:34:02 [2ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:34:02 [3ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:34:02 [3ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x23d01d0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:34:02 [3ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3530. 2023-02-19 03:34:02 [3ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:34:02 [3ms] [Info] [carb] [3530] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:34:02 [3ms] [Info] [carb.launcher.plugin] [3547] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3548 from the parent 3530. 2023-02-19 03:34:02 [3ms] [Info] [carb.launcher.plugin] [3547] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3548. 2023-02-19 03:34:02 [4ms] [Info] [carb.launcher.plugin] [3547] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3548. 2023-02-19 03:34:02 [4ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3548 from the parent 3530. 2023-02-19 03:34:02 [4ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3548 {process = 0x23d01d0} 2023-02-19 03:34:02 [4ms] [Info] [carb.launcher.plugin] [3549] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3548. 2023-02-19 03:34:02 [4ms] [Info] [omni.core] [3530] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:34:02 [4ms] [Info] [omni.core] [3530] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:34:02 [4ms] [Info] [carb.launcher.plugin] [3550] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3548. 2023-02-19 03:34:02 [339ms] [Error] [omni.core] [3550] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:34:02 [344ms] [Error] [omni.core] [3550] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:34:02,953 - system_log - INFO - reconnecting ... 2023-02-19 03:34:02 [354ms] [Error] [omni.core] [3550] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:34:02,963 - system_log - INFO - connected 2023-02-19 03:34:02 [355ms] [Error] [omni.core] [3550] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [3548] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 03:34:02 [388ms] [Info] [carb.launcher.plugin] [3550] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:34:02 [388ms] [Info] [carb.launcher.plugin] [3549] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:34:02 [388ms] [Info] [carb.launcher.plugin] [3550] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3548, bytes = 0} 2023-02-19 03:34:02 [388ms] [Info] [carb.launcher.plugin] [3549] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3548, bytes = 0} 2023-02-19 03:34:02 [388ms] [Info] [carb.launcher.plugin] [3550] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 3548. 2023-02-19 03:34:02 [388ms] [Info] [carb.launcher.plugin] [3549] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 3548. 2023-02-19 03:34:03 [115,057ms] [Info] [omni.server.app] [3092] ../../../source/apps/omni.server/main.cpp:main():106: Got signal to exit the application. 2023-02-19 03:34:03 [115,057ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():391: OmniverseServer shutdown process starting. 2023-02-19 03:34:03 [115,057ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():392: Waiting for any outstanding transactions to complete. 2023-02-19 03:34:03 [115,057ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():396: Releasing WebSocket listen ports 2023-02-19 03:34:03 [115,057ms] [Info] [omni.server] [3600] ../../../source/plugins/omni.server/ApiConnection.cpp:shutdownCore():70: Shutdown Core 2023-02-19 03:34:03 [115,058ms] [Error] [omni.core] [3107] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Shutting down server...wait 2023-02-19 03:34:03 [115,063ms] [Info] [carb.launcher.plugin] [3134] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:34:03 [115,063ms] [Info] [carb.launcher.plugin] [3134] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3133, bytes = 0} 2023-02-19 03:34:03 [115,063ms] [Info] [carb.launcher.plugin] [3135] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:34:03 [115,063ms] [Info] [carb.launcher.plugin] [3134] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 3133. 2023-02-19 03:34:03 [115,063ms] [Info] [carb.launcher.plugin] [3135] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3133, bytes = 0} 2023-02-19 03:34:03 [115,063ms] [Info] [carb.launcher.plugin] [3135] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 3133. 2023-02-19 03:34:03 [115,084ms] [Info] [omni.core] [3124] ../../../source/plugins/omni.core/impl/Scheduler.cpp:scheduler():122: Scheduling task is shutted down... 2023-02-19 03:34:03 [115,084ms] [Info] [omni.core] [3098] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():422: Finished content remove task 2023-02-19 03:34:03 [115,084ms] [Warning] [carb] [3600] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 03:34:03 [115,085ms] [Info] [omni.core] [3600] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:operator()():849: Content resolver statistics: Total: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0|Temporal: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0|Persistent: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0 2023-02-19 03:34:03 [115,085ms] [Info] [omni.server.app] [3092] ../../../source/apps/omni.server/main.cpp:main():120: Releasing the OmniverseServer instance. 2023-02-19 03:34:03 [115,085ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():310: OmniverseServerImpl destructor. 2023-02-19 03:34:03 [115,085ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():312: Releasing ClientConnectionManager 2023-02-19 03:34:03 [115,085ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():315: Releasing ServiceConnectionManager 2023-02-19 03:34:03 [115,085ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():318: Releasing LftConnectionManager 2023-02-19 03:34:03 [115,085ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():321: Releasing ConnectionManager 2023-02-19 03:34:03 [115,085ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 3133 {exitCode = 137, flags = 0x00000002, timeout = 5000} 2023-02-19 03:34:03 [115,086ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x7f4ec40013b0 (3133) to exit {timeout = 18446744073709551615} 2023-02-19 03:34:03 [115,086ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 03:34:03 [115,086ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x7f4ec40013b0 {pid = 3133} 2023-02-19 03:34:03 [115,086ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x7f4ec40013b0 {pid = 3133} 2023-02-19 03:34:03 [115,086ms] [Info] [omni.core] [3098] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():457: discovery client: exiting 2023-02-19 03:34:03 [115,087ms] [Info] [omni.core] [3092] ../../../source/plugins/common/ChildProcess.cpp:kill():125: Killing child server process ./omni-lft-service/omni-lft-service 2023-02-19 03:34:03 [115,087ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 3105 {exitCode = 137, flags = 0x00000003, timeout = 5000} 2023-02-19 03:34:03 [115,088ms] [Info] [carb.launcher.plugin] [3106] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:34:03 [115,088ms] [Info] [carb.launcher.plugin] [3107] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:34:03 [115,088ms] [Info] [carb.launcher.plugin] [3106] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3105, bytes = 0} 2023-02-19 03:34:03 [115,088ms] [Info] [carb.launcher.plugin] [3107] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3105, bytes = 0} 2023-02-19 03:34:03 [115,088ms] [Info] [carb.launcher.plugin] [3106] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 3105. 2023-02-19 03:34:03 [115,088ms] [Info] [carb.launcher.plugin] [3107] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 3105. 2023-02-19 03:34:03 [115,090ms] [Info] [omni.core] [3092] ../../../source/plugins/common/ChildProcess.cpp:kill():127: Waiting for child server process ./omni-lft-service/omni-lft-service to exit 2023-02-19 03:34:03 [115,090ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x18fe140 (3105) to exit {timeout = 18446744073709551615} 2023-02-19 03:34:03 [115,090ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 03:34:03 [115,090ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x18fe140 {pid = 3105} 2023-02-19 03:34:03 [115,090ms] [Info] [carb.launcher.plugin] [3092] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x18fe140 {pid = 3105} 2023-02-19 03:34:03 [115,090ms] [Info] [omni.server] [3092] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():327: Destructor complete. 2023-02-19 03:34:04 [2,004ms] [Info] [omni.core] [3539] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 03:34:04 [2,004ms] [Info] [omni.core] [3539] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:34:04 [2,004ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:34:04 [2,005ms] [Info] [omni.core] [3530] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 03:34:04 [2,006ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:34:04 [2,007ms] [Info] [omni.core] [3539] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:34:04 [2,030ms] [Warning] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 03:34:04 [2,030ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:34:04 [2,044ms] [Warning] [carb] [3530] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 03:34:04 [2,044ms] [Info] [carb] [3530] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 7 records 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 7 records 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 7 records 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-02-19 03:34:04 [2,045ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-02-19 03:34:04 [2,046ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-02-19 03:34:04 [2,046ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-02-19 03:34:04 [2,046ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-02-19 03:34:04 [2,046ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-02-19 03:34:04 [2,046ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-02-19 03:34:04 [2,046ms] [Info] [omni.meta-upgrade] [3530] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-02-19 03:34:04 [2,047ms] [Info] [omni.core] [3530] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-02-19 03:34:04 [2,049ms] [Info] [omni.core] [3530] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-02-19 03:34:04 [2,055ms] [Info] [omni.core] [3530] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-02-19 03:34:04 [2,055ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-02-19 03:34:04 [2,055ms] [Info] [omni.core] [3540] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-02-19 03:34:04 [2,057ms] [Info] [omni.core] [3540] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-02-19 03:34:04 [2,057ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7f56b80048c0 for './omni.delta-applier.app'. 2023-02-19 03:34:04 [2,057ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 3530. 2023-02-19 03:34:04 [2,058ms] [Info] [carb.launcher.plugin] [3547] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3772 from the parent 3530. 2023-02-19 03:34:04 [2,058ms] [Info] [carb.launcher.plugin] [3547] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3772. 2023-02-19 03:34:04 [2,061ms] [Info] [carb.launcher.plugin] [3547] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3772. 2023-02-19 03:34:04 [2,061ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3772 from the parent 3530. 2023-02-19 03:34:04 [2,061ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3772 {process = 0x7f56b80048c0} 2023-02-19 03:34:04 [2,061ms] [Info] [carb.launcher.plugin] [3774] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3772. 2023-02-19 03:34:04 [2,061ms] [Info] [carb.launcher.plugin] [3773] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3772. 2023-02-19 03:34:04 [2,264ms] [Info] [omni.core] [3542] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-02-19 03:34:06 [4,055ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-02-19 03:34:06 [4,062ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-02-19 03:34:06 [4,062ms] [Info] [omni.signalhandler] [3530] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-02-19 03:34:06 [4,062ms] [Info] [omni.signalhandler] [3530] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-02-19 03:34:06 [4,062ms] [Info] [omni.signalhandler] [3530] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-02-19 03:34:14 [12,004ms] [Info] [omni.core] [3540] ../../../source/plugins/common/ChildProcess.cpp:kill():125: Killing child server process ./omni-lft-service/omni-lft-service 2023-02-19 03:34:14 [12,004ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 3548 {exitCode = 137, flags = 0x00000003, timeout = 5000} 2023-02-19 03:34:14 [12,006ms] [Info] [omni.core] [3540] ../../../source/plugins/common/ChildProcess.cpp:kill():127: Waiting for child server process ./omni-lft-service/omni-lft-service to exit 2023-02-19 03:34:14 [12,006ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x23d01d0 (3548) to exit {timeout = 18446744073709551615} 2023-02-19 03:34:14 [12,006ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 03:34:14 [12,006ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x23d01d0 {pid = 3548} 2023-02-19 03:34:14 [12,006ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x23d01d0 {pid = 3548} 2023-02-19 03:34:14 [12,006ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x23d01d0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:34:14 [12,006ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3530. 2023-02-19 03:34:14 [12,007ms] [Info] [carb.launcher.plugin] [3547] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3848 from the parent 3530. 2023-02-19 03:34:14 [12,007ms] [Info] [carb.launcher.plugin] [3547] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3848. 2023-02-19 03:34:14 [12,010ms] [Info] [carb.launcher.plugin] [3547] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3848. 2023-02-19 03:34:14 [12,010ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3848 from the parent 3530. 2023-02-19 03:34:14 [12,010ms] [Info] [carb.launcher.plugin] [3540] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3848 {process = 0x23d01d0} 2023-02-19 03:34:14 [12,010ms] [Info] [carb.launcher.plugin] [3849] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3848. 2023-02-19 03:34:14 [12,011ms] [Info] [carb.launcher.plugin] [3850] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3848. 2023-02-19 03:34:14 [12,010ms] [Info] [omni.core] [3540] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:34:14 [12,011ms] [Info] [omni.core] [3540] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:34:14 [12,265ms] [Error] [omni.core] [3850] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:34:14 [12,269ms] [Error] [omni.core] [3850] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:34:14,878 - system_log - INFO - reconnecting ... 2023-02-19 03:34:14 [12,279ms] [Error] [omni.core] [3850] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:34:14,888 - system_log - INFO - connected 2023-02-19 03:34:14 [12,280ms] [Error] [omni.core] [3850] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-02-19 03:34:44 [0ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4484] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4484] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4484] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4484] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4484] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4484] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4484] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4484] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4484] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:34:44 [2ms] [Info] [omni.server] [4484] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:34:44 [2ms] [Info] [omni.server] [4484] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "" } } } 2023-02-19 03:34:44 [2ms] [Info] [carb] [4484] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:34:44 [2ms] [Info] [carb] [4484] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:34:44 [3ms] [Info] [carb] [4484] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:34:44 [4ms] [Info] [carb] [4484] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:34:44 [4ms] [Info] [carb.launcher.plugin] [4484] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x18823b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:34:44 [4ms] [Info] [carb.launcher.plugin] [4484] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 4484. 2023-02-19 03:34:44 [4ms] [Info] [carb] [4484] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:34:44 [4ms] [Info] [carb] [4484] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:34:44 [4ms] [Info] [carb.launcher.plugin] [4507] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4509 from the parent 4484. 2023-02-19 03:34:44 [4ms] [Info] [carb.launcher.plugin] [4507] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4509. 2023-02-19 03:34:44 [5ms] [Info] [carb.launcher.plugin] [4507] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4509. 2023-02-19 03:34:44 [5ms] [Info] [carb.launcher.plugin] [4484] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4509 from the parent 4484. 2023-02-19 03:34:44 [5ms] [Info] [carb.launcher.plugin] [4484] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4509 {process = 0x18823b0} 2023-02-19 03:34:44 [5ms] [Info] [omni.core] [4484] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:34:44 [5ms] [Info] [omni.core] [4484] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:34:44 [5ms] [Info] [carb.launcher.plugin] [4512] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4509. 2023-02-19 03:34:44 [5ms] [Info] [carb.launcher.plugin] [4513] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4509. 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:34:44 [0ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4519] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:34:44 [1ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4519] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4519] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4519] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4519] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4519] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4519] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4519] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4519] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:34:44 [1ms] [Info] [omni.server] [4519] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:34:44 [2ms] [Info] [omni.server] [4519] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 03:34:44 [2ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:34:44 [2ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:34:44 [2ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:34:44 [3ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:34:44 [3ms] [Info] [carb.launcher.plugin] [4519] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x27593b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:34:44 [3ms] [Info] [carb.launcher.plugin] [4519] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 4519. 2023-02-19 03:34:44 [3ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:34:44 [3ms] [Info] [carb] [4519] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:34:44 [3ms] [Info] [carb.launcher.plugin] [4531] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4532 from the parent 4519. 2023-02-19 03:34:44 [3ms] [Info] [carb.launcher.plugin] [4531] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4532. 2023-02-19 03:34:44 [4ms] [Info] [carb.launcher.plugin] [4531] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4532. 2023-02-19 03:34:44 [5ms] [Info] [carb.launcher.plugin] [4519] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4532 from the parent 4519. 2023-02-19 03:34:44 [5ms] [Info] [carb.launcher.plugin] [4533] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4532. 2023-02-19 03:34:44 [5ms] [Info] [carb.launcher.plugin] [4519] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4532 {process = 0x27593b0} 2023-02-19 03:34:44 [5ms] [Info] [omni.core] [4519] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:34:44 [5ms] [Info] [omni.core] [4519] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:34:44 [5ms] [Info] [carb.launcher.plugin] [4534] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4532. 2023-02-19 03:34:45 [287ms] [Error] [omni.core] [4534] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:34:45 [291ms] [Error] [omni.core] [4534] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:34:45,070 - system_log - INFO - reconnecting ... 2023-02-19 03:34:45 [302ms] [Error] [omni.core] [4534] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:34:45,081 - system_log - INFO - connected 2023-02-19 03:34:45 [302ms] [Error] [omni.core] [4534] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [4532] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 03:34:45 [333ms] [Info] [carb.launcher.plugin] [4533] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:34:45 [333ms] [Info] [carb.launcher.plugin] [4533] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 4532, bytes = 0} 2023-02-19 03:34:45 [333ms] [Info] [carb.launcher.plugin] [4533] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 4532. 2023-02-19 03:34:45 [333ms] [Info] [carb.launcher.plugin] [4534] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:34:45 [333ms] [Info] [carb.launcher.plugin] [4534] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 4532, bytes = 0} 2023-02-19 03:34:45 [333ms] [Info] [carb.launcher.plugin] [4534] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 4532. 2023-02-19 03:34:46 [2,005ms] [Info] [omni.core] [4526] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 03:34:46 [2,005ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:34:46 [2,005ms] [Info] [omni.core] [4526] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:34:46 [2,008ms] [Info] [omni.core] [4519] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 03:34:46 [2,008ms] [Info] [carb] [4519] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:34:46 [2,008ms] [Warning] [omni.keyvaluedatabase.leveldb] [4519] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 03:34:46 [2,009ms] [Info] [omni.core] [4526] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:34:46 [2,019ms] [Warning] [omni.keyvaluedatabase.leveldb] [4519] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 03:34:46 [2,019ms] [Fatal] [omni.core] [4519] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-02-19 03:35:09 [0ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4680] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:35:09 [1ms] [Info] [omni.server] [4680] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:35:09 [1ms] [Info] [omni.server] [4680] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:35:09 [1ms] [Info] [omni.server] [4680] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:35:09 [1ms] [Info] [omni.server] [4680] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:35:09 [2ms] [Info] [omni.server] [4680] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:35:09 [2ms] [Info] [omni.server] [4680] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:35:09 [2ms] [Info] [omni.server] [4680] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:35:09 [2ms] [Info] [omni.server] [4680] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:35:09 [2ms] [Info] [omni.server] [4680] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:35:09 [2ms] [Info] [omni.server] [4680] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "" } } } 2023-02-19 03:35:09 [2ms] [Info] [carb] [4680] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4680] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4680] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:35:09 [3ms] [Info] [carb] [4680] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:35:09 [3ms] [Info] [carb.launcher.plugin] [4680] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xd2d3b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:35:09 [3ms] [Info] [carb.launcher.plugin] [4680] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 4680. 2023-02-19 03:35:09 [3ms] [Info] [carb] [4680] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:35:09 [3ms] [Info] [carb] [4680] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:35:09 [3ms] [Info] [carb.launcher.plugin] [4703] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4704 from the parent 4680. 2023-02-19 03:35:09 [3ms] [Info] [carb.launcher.plugin] [4703] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4704. 2023-02-19 03:35:09 [4ms] [Info] [carb.launcher.plugin] [4703] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4704. 2023-02-19 03:35:09 [4ms] [Info] [carb.launcher.plugin] [4680] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4704 from the parent 4680. 2023-02-19 03:35:09 [4ms] [Info] [carb.launcher.plugin] [4680] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4704 {process = 0xd2d3b0} 2023-02-19 03:35:09 [4ms] [Info] [omni.core] [4680] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:35:09 [4ms] [Info] [omni.core] [4680] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:35:09 [4ms] [Info] [carb.launcher.plugin] [4707] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4704. 2023-02-19 03:35:09 [4ms] [Info] [carb.launcher.plugin] [4708] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4704. 2023-02-19 03:35:09 [0ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:35:09 [0ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:35:09 [1ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4715] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:35:09 [2ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:35:09 [3ms] [Info] [omni.server] [4715] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:35:09 [3ms] [Info] [omni.server] [4715] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:35:09 [3ms] [Info] [omni.server] [4715] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:35:09 [3ms] [Info] [omni.server] [4715] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:35:09 [3ms] [Info] [omni.server] [4715] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:35:09 [3ms] [Info] [omni.server] [4715] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:35:09 [3ms] [Info] [omni.server] [4715] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:35:09 [3ms] [Info] [omni.server] [4715] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:35:09 [3ms] [Info] [omni.server] [4715] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:35:09 [3ms] [Info] [omni.server] [4715] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 03:35:09 [3ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:35:09 [4ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:35:09 [4ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:35:09 [5ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:35:09 [5ms] [Info] [carb.launcher.plugin] [4715] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x10e53b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:35:09 [5ms] [Info] [carb.launcher.plugin] [4715] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 4715. 2023-02-19 03:35:09 [5ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:35:09 [5ms] [Info] [carb] [4715] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:35:09 [5ms] [Info] [carb.launcher.plugin] [4727] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4728 from the parent 4715. 2023-02-19 03:35:09 [5ms] [Info] [carb.launcher.plugin] [4727] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4728. 2023-02-19 03:35:09 [6ms] [Info] [carb.launcher.plugin] [4727] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4728. 2023-02-19 03:35:09 [6ms] [Info] [carb.launcher.plugin] [4715] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4728 from the parent 4715. 2023-02-19 03:35:09 [6ms] [Info] [carb.launcher.plugin] [4729] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4728. 2023-02-19 03:35:09 [6ms] [Info] [carb.launcher.plugin] [4715] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4728 {process = 0x10e53b0} 2023-02-19 03:35:09 [6ms] [Info] [omni.core] [4715] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:35:09 [6ms] [Info] [omni.core] [4715] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:35:09 [6ms] [Info] [carb.launcher.plugin] [4730] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4728. 2023-02-19 03:35:10 [291ms] [Error] [omni.core] [4730] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:35:10 [295ms] [Error] [omni.core] [4730] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:35:10,148 - system_log - INFO - reconnecting ... 2023-02-19 03:35:10 [305ms] [Error] [omni.core] [4730] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:35:10,158 - system_log - INFO - connected 2023-02-19 03:35:10 [306ms] [Error] [omni.core] [4730] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [4728] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 03:35:10 [334ms] [Info] [carb.launcher.plugin] [4730] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:35:10 [334ms] [Info] [carb.launcher.plugin] [4730] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 4728, bytes = 0} 2023-02-19 03:35:10 [334ms] [Info] [carb.launcher.plugin] [4730] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 4728. 2023-02-19 03:35:10 [334ms] [Info] [carb.launcher.plugin] [4729] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:35:10 [334ms] [Info] [carb.launcher.plugin] [4729] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 4728, bytes = 0} 2023-02-19 03:35:10 [334ms] [Info] [carb.launcher.plugin] [4729] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 4728. 2023-02-19 03:35:11 [2,007ms] [Info] [omni.core] [4721] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 03:35:11 [2,007ms] [Info] [omni.core] [4721] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:35:11 [2,007ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:35:11 [2,010ms] [Info] [omni.core] [4715] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 03:35:11 [2,010ms] [Info] [carb] [4715] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:35:11 [2,011ms] [Warning] [omni.keyvaluedatabase.leveldb] [4715] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 03:35:11 [2,011ms] [Info] [omni.core] [4721] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:35:11 [2,018ms] [Warning] [omni.keyvaluedatabase.leveldb] [4715] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 03:35:11 [2,018ms] [Fatal] [omni.core] [4715] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-02-19 03:46:08 [0ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:46:08 [0ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:46:08 [0ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 03:46:08 [0ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:46:08 [0ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:46:08 [0ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:46:08 [0ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 03:46:08 [0ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:46:08 [0ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:46:08 [0ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 03:46:08 [1ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 03:46:08 [1ms] [Info] [omni.server] [10117] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 03:46:08 [1ms] [Info] [omni.server] [10117] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 03:46:08 [1ms] [Info] [omni.server] [10117] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 03:46:08 [1ms] [Info] [omni.server] [10117] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 03:46:08 [1ms] [Info] [omni.server] [10117] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 03:46:08 [1ms] [Info] [omni.server] [10117] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 03:46:08 [1ms] [Info] [omni.server] [10117] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 03:46:08 [1ms] [Info] [omni.server] [10117] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 03:46:08 [2ms] [Info] [omni.server] [10117] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 03:46:08 [2ms] [Info] [omni.server] [10117] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 03:46:08 [2ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 03:46:08 [2ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 03:46:08 [2ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 03:46:08 [3ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 03:46:08 [3ms] [Info] [carb.launcher.plugin] [10117] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xb7c1d0 for './omni-lft-service/omni-lft-service'. 2023-02-19 03:46:08 [3ms] [Info] [carb.launcher.plugin] [10117] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 10117. 2023-02-19 03:46:08 [3ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 03:46:08 [3ms] [Info] [carb] [10117] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 03:46:08 [3ms] [Info] [carb.launcher.plugin] [10132] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 10133 from the parent 10117. 2023-02-19 03:46:08 [3ms] [Info] [carb.launcher.plugin] [10132] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 10133. 2023-02-19 03:46:08 [4ms] [Info] [carb.launcher.plugin] [10132] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 10133. 2023-02-19 03:46:08 [4ms] [Info] [carb.launcher.plugin] [10117] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 10133 from the parent 10117. 2023-02-19 03:46:08 [4ms] [Info] [carb.launcher.plugin] [10117] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 10133 {process = 0xb7c1d0} 2023-02-19 03:46:08 [4ms] [Info] [omni.core] [10117] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 03:46:08 [4ms] [Info] [omni.core] [10117] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 03:46:08 [4ms] [Info] [carb.launcher.plugin] [10135] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 10133. 2023-02-19 03:46:08 [4ms] [Info] [carb.launcher.plugin] [10134] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 10133. 2023-02-19 03:46:09 [263ms] [Error] [omni.core] [10135] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 03:46:09 [267ms] [Error] [omni.core] [10135] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:46:09,050 - system_log - INFO - reconnecting ... 2023-02-19 03:46:09 [277ms] [Error] [omni.core] [10135] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 20:46:09,059 - system_log - INFO - connected 2023-02-19 03:46:09 [277ms] [Error] [omni.core] [10135] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [10133] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 03:46:09 [305ms] [Info] [carb.launcher.plugin] [10134] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:46:09 [305ms] [Info] [carb.launcher.plugin] [10135] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 03:46:09 [305ms] [Info] [carb.launcher.plugin] [10134] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 10133, bytes = 0} 2023-02-19 03:46:09 [305ms] [Info] [carb.launcher.plugin] [10135] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 10133, bytes = 0} 2023-02-19 03:46:09 [305ms] [Info] [carb.launcher.plugin] [10134] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 10133. 2023-02-19 03:46:09 [305ms] [Info] [carb.launcher.plugin] [10135] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 10133. 2023-02-19 03:46:10 [2,004ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 03:46:10 [2,004ms] [Info] [omni.core] [10122] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 03:46:10 [2,004ms] [Info] [omni.core] [10122] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 03:46:10 [2,005ms] [Info] [omni.core] [10117] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 03:46:10 [2,005ms] [Info] [carb] [10117] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 03:46:10 [2,005ms] [Warning] [omni.keyvaluedatabase.leveldb] [10117] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 03:46:10 [2,005ms] [Info] [omni.core] [10122] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 03:46:10 [2,010ms] [Warning] [omni.keyvaluedatabase.leveldb] [10117] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 03:46:10 [2,010ms] [Fatal] [omni.core] [10117] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-02-19 04:11:02 [0ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 04:11:02 [0ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 04:11:02 [0ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 04:11:02 [1ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 04:11:02 [2ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 04:11:02 [2ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 04:11:02 [3ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 04:11:02 [3ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 04:11:02 [3ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 04:11:02 [4ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 04:11:02 [4ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 04:11:02 [5ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 04:11:02 [5ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 04:11:02 [6ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 04:11:02 [6ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 04:11:02 [6ms] [Info] [carb] [24297] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 04:11:02 [6ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 04:11:02 [6ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 04:11:02 [6ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 04:11:02 [6ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 04:11:02 [6ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 04:11:02 [7ms] [Info] [omni.server] [24297] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 04:11:02 [7ms] [Info] [omni.server] [24297] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 04:11:02 [7ms] [Info] [omni.server] [24297] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 04:11:02 [7ms] [Info] [omni.server] [24297] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 04:11:02 [7ms] [Info] [omni.server] [24297] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 04:11:02 [7ms] [Info] [omni.server] [24297] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 04:11:02 [7ms] [Info] [omni.server] [24297] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 04:11:02 [7ms] [Info] [omni.server] [24297] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 04:11:02 [7ms] [Info] [omni.server] [24297] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 04:11:02 [7ms] [Info] [omni.server] [24297] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 04:11:02 [7ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 04:11:02 [7ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 04:11:02 [8ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 04:11:02 [8ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 04:11:02 [8ms] [Info] [carb.launcher.plugin] [24297] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x18f41c0 for './omni-lft-service/omni-lft-service'. 2023-02-19 04:11:02 [8ms] [Info] [carb.launcher.plugin] [24297] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 24297. 2023-02-19 04:11:02 [8ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 04:11:02 [8ms] [Info] [carb] [24297] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 04:11:02 [9ms] [Info] [carb.launcher.plugin] [24309] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 24310 from the parent 24297. 2023-02-19 04:11:02 [9ms] [Info] [carb.launcher.plugin] [24309] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 24310. 2023-02-19 04:11:02 [9ms] [Info] [carb.launcher.plugin] [24309] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 24310. 2023-02-19 04:11:02 [10ms] [Info] [carb.launcher.plugin] [24297] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 24310 from the parent 24297. 2023-02-19 04:11:02 [10ms] [Info] [carb.launcher.plugin] [24297] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 24310 {process = 0x18f41c0} 2023-02-19 04:11:02 [10ms] [Info] [omni.core] [24297] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 04:11:02 [10ms] [Info] [omni.core] [24297] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 04:11:02 [10ms] [Info] [carb.launcher.plugin] [24311] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 24310. 2023-02-19 04:11:02 [10ms] [Info] [carb.launcher.plugin] [24312] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 24310. 2023-02-19 04:11:03 [267ms] [Error] [omni.core] [24312] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 04:11:03 [271ms] [Error] [omni.core] [24312] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 21:11:03,145 - system_log - INFO - reconnecting ... 2023-02-19 04:11:03 [282ms] [Error] [omni.core] [24312] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 21:11:03,156 - system_log - INFO - connected 2023-02-19 04:11:03 [283ms] [Error] [omni.core] [24312] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [24310] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 04:11:03 [308ms] [Info] [carb.launcher.plugin] [24312] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 04:11:03 [308ms] [Info] [carb.launcher.plugin] [24312] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 24310, bytes = 0} 2023-02-19 04:11:03 [308ms] [Info] [carb.launcher.plugin] [24312] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 24310. 2023-02-19 04:11:03 [308ms] [Info] [carb.launcher.plugin] [24311] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 04:11:03 [308ms] [Info] [carb.launcher.plugin] [24311] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 24310, bytes = 0} 2023-02-19 04:11:03 [308ms] [Info] [carb.launcher.plugin] [24311] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 24310. 2023-02-19 04:11:04 [2,010ms] [Info] [omni.core] [24302] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 04:11:04 [2,010ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 04:11:04 [2,010ms] [Info] [omni.core] [24302] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 04:11:04 [2,014ms] [Info] [omni.core] [24297] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 04:11:04 [2,014ms] [Info] [carb] [24297] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 04:11:04 [2,014ms] [Warning] [omni.keyvaluedatabase.leveldb] [24297] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 04:11:04 [2,015ms] [Info] [omni.core] [24302] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 04:11:04 [2,017ms] [Warning] [omni.keyvaluedatabase.leveldb] [24297] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 04:11:04 [2,017ms] [Fatal] [omni.core] [24297] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-02-19 04:11:46 [0ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 04:11:46 [0ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 04:11:46 [0ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 04:11:46 [0ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 04:11:46 [0ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 04:11:46 [0ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 04:11:46 [0ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 04:11:46 [0ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 04:11:46 [0ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 04:11:46 [1ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 04:11:46 [2ms] [Info] [omni.server] [24689] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 04:11:46 [2ms] [Info] [omni.server] [24689] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 04:11:46 [2ms] [Info] [omni.server] [24689] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 04:11:46 [2ms] [Info] [omni.server] [24689] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 04:11:46 [2ms] [Info] [omni.server] [24689] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 04:11:46 [2ms] [Info] [omni.server] [24689] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 04:11:46 [2ms] [Info] [omni.server] [24689] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 04:11:46 [2ms] [Info] [omni.server] [24689] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 04:11:46 [2ms] [Info] [omni.server] [24689] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 04:11:46 [2ms] [Info] [omni.server] [24689] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 04:11:46 [2ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 04:11:46 [2ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 04:11:46 [3ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 04:11:46 [3ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 04:11:46 [3ms] [Info] [carb.launcher.plugin] [24689] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x26fb1c0 for './omni-lft-service/omni-lft-service'. 2023-02-19 04:11:46 [3ms] [Info] [carb.launcher.plugin] [24689] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 24689. 2023-02-19 04:11:46 [3ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 04:11:46 [3ms] [Info] [carb] [24689] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 04:11:46 [3ms] [Info] [carb.launcher.plugin] [24701] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 24703 from the parent 24689. 2023-02-19 04:11:46 [3ms] [Info] [carb.launcher.plugin] [24701] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 24703. 2023-02-19 04:11:46 [4ms] [Info] [carb.launcher.plugin] [24701] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 24703. 2023-02-19 04:11:46 [4ms] [Info] [carb.launcher.plugin] [24689] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 24703 from the parent 24689. 2023-02-19 04:11:46 [4ms] [Info] [carb.launcher.plugin] [24689] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 24703 {process = 0x26fb1c0} 2023-02-19 04:11:46 [4ms] [Info] [omni.core] [24689] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 04:11:46 [4ms] [Info] [carb.launcher.plugin] [24704] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 24703. 2023-02-19 04:11:46 [4ms] [Info] [carb.launcher.plugin] [24705] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 24703. 2023-02-19 04:11:46 [4ms] [Info] [omni.core] [24689] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 04:11:46 [251ms] [Error] [omni.core] [24705] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 04:11:46 [255ms] [Error] [omni.core] [24705] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 21:11:46,536 - system_log - INFO - reconnecting ... 2023-02-19 04:11:46 [264ms] [Error] [omni.core] [24705] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 21:11:46,545 - system_log - INFO - connected 2023-02-19 04:11:46 [265ms] [Error] [omni.core] [24705] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [24703] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 04:11:46 [290ms] [Info] [carb.launcher.plugin] [24705] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 04:11:46 [290ms] [Info] [carb.launcher.plugin] [24704] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 04:11:46 [291ms] [Info] [carb.launcher.plugin] [24704] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 24703, bytes = 0} 2023-02-19 04:11:46 [290ms] [Info] [carb.launcher.plugin] [24705] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 24703, bytes = 0} 2023-02-19 04:11:46 [291ms] [Info] [carb.launcher.plugin] [24704] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 24703. 2023-02-19 04:11:46 [291ms] [Info] [carb.launcher.plugin] [24705] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 24703. 2023-02-19 04:11:48 [2,005ms] [Info] [omni.core] [24696] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 04:11:48 [2,005ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 04:11:48 [2,005ms] [Info] [omni.core] [24696] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 04:11:48 [2,005ms] [Info] [omni.core] [24689] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 04:11:48 [2,005ms] [Info] [carb] [24689] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 04:11:48 [2,005ms] [Warning] [omni.keyvaluedatabase.leveldb] [24689] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 04:11:48 [2,006ms] [Info] [omni.core] [24696] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 04:11:48 [2,013ms] [Warning] [omni.keyvaluedatabase.leveldb] [24689] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 04:11:48 [2,013ms] [Fatal] [omni.core] [24689] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-02-19 05:01:55 [5,272,550ms] [Info] [omni.server.app] [3530] ../../../source/apps/omni.server/main.cpp:main():106: Got signal to exit the application. 2023-02-19 05:01:55 [5,272,550ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():391: OmniverseServer shutdown process starting. 2023-02-19 05:01:55 [5,272,550ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():392: Waiting for any outstanding transactions to complete. 2023-02-19 05:01:55 [5,272,550ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():396: Releasing WebSocket listen ports 2023-02-19 05:01:55 [5,272,550ms] [Info] [omni.server] [49953] ../../../source/plugins/omni.server/ApiConnection.cpp:shutdownCore():70: Shutdown Core 2023-02-19 05:01:55 [5,272,553ms] [Info] [omni.core] [3546] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:operator()():426: discovery client: received non-ok status from the discovery service: 5 / Connection is terminated 2023-02-19 05:01:55 [5,272,553ms] [Info] [omni.core] [3762] ../../../source/plugins/omni.core/impl/Scheduler.cpp:scheduler():122: Scheduling task is shutted down... 2023-02-19 05:01:55 [5,272,555ms] [Error] [omni.core] [3850] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Shutting down server...wait 2023-02-19 05:01:55 [5,272,555ms] [Info] [omni.core] [3541] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():422: Finished content remove task 2023-02-19 05:01:55 [5,272,555ms] [Warning] [carb] [49953] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 05:01:55 [5,272,555ms] [Info] [omni.core] [49953] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:operator()():849: Content resolver statistics: Total: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0|Temporal: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0|Persistent: flush: 0 write: 0 read: 0 stat: 0 remove: 0 list: 0 2023-02-19 05:01:55 [5,272,556ms] [Info] [omni.server.app] [3530] ../../../source/apps/omni.server/main.cpp:main():120: Releasing the OmniverseServer instance. 2023-02-19 05:01:55 [5,272,556ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():310: OmniverseServerImpl destructor. 2023-02-19 05:01:55 [5,272,556ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():312: Releasing ClientConnectionManager 2023-02-19 05:01:55 [5,272,556ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():315: Releasing ServiceConnectionManager 2023-02-19 05:01:55 [5,272,556ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():318: Releasing LftConnectionManager 2023-02-19 05:01:55 [5,272,556ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():321: Releasing ConnectionManager 2023-02-19 05:01:55 [5,272,556ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 3772 {exitCode = 137, flags = 0x00000002, timeout = 5000} 2023-02-19 05:01:55 [5,272,557ms] [Info] [carb.launcher.plugin] [3774] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 05:01:55 [5,272,557ms] [Info] [carb.launcher.plugin] [3773] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 05:01:55 [5,272,558ms] [Info] [carb.launcher.plugin] [3774] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3772, bytes = 0} 2023-02-19 05:01:55 [5,272,558ms] [Info] [carb.launcher.plugin] [3773] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3772, bytes = 0} 2023-02-19 05:01:55 [5,272,558ms] [Info] [carb.launcher.plugin] [3774] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 3772. 2023-02-19 05:01:55 [5,272,558ms] [Info] [carb.launcher.plugin] [3773] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 3772. 2023-02-19 05:01:55 [5,272,559ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x7f56b80048c0 (3772) to exit {timeout = 18446744073709551615} 2023-02-19 05:01:55 [5,272,559ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 05:01:55 [5,272,559ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x7f56b80048c0 {pid = 3772} 2023-02-19 05:01:55 [5,272,560ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x7f56b80048c0 {pid = 3772} 2023-02-19 05:01:55 [5,272,560ms] [Info] [omni.core] [3539] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():457: discovery client: exiting 2023-02-19 05:01:55 [5,272,560ms] [Info] [omni.core] [3530] ../../../source/plugins/common/ChildProcess.cpp:kill():125: Killing child server process ./omni-lft-service/omni-lft-service 2023-02-19 05:01:55 [5,272,560ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 3848 {exitCode = 137, flags = 0x00000003, timeout = 5000} 2023-02-19 05:01:55 [5,272,563ms] [Info] [carb.launcher.plugin] [3850] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 05:01:55 [5,272,563ms] [Info] [carb.launcher.plugin] [3850] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3848, bytes = 0} 2023-02-19 05:01:55 [5,272,563ms] [Info] [carb.launcher.plugin] [3850] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 3848. 2023-02-19 05:01:55 [5,272,563ms] [Info] [carb.launcher.plugin] [3849] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 05:01:55 [5,272,563ms] [Info] [carb.launcher.plugin] [3849] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3848, bytes = 0} 2023-02-19 05:01:55 [5,272,563ms] [Info] [carb.launcher.plugin] [3849] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 3848. 2023-02-19 05:01:55 [5,272,565ms] [Info] [omni.core] [3530] ../../../source/plugins/common/ChildProcess.cpp:kill():127: Waiting for child server process ./omni-lft-service/omni-lft-service to exit 2023-02-19 05:01:55 [5,272,565ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x23d01d0 (3848) to exit {timeout = 18446744073709551615} 2023-02-19 05:01:55 [5,272,565ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-02-19 05:01:55 [5,272,565ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x23d01d0 {pid = 3848} 2023-02-19 05:01:55 [5,272,565ms] [Info] [carb.launcher.plugin] [3530] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x23d01d0 {pid = 3848} 2023-02-19 05:01:55 [5,272,566ms] [Info] [omni.server] [3530] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():327: Destructor complete. 2023-02-19 05:16:28 [0ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:28 [1ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:16:28 [3ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 05:16:28 [7ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:28 [14ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:28 [14ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:28 [17ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 05:16:28 [18ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:28 [18ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:28 [19ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:28 [20ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:16:28 [20ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:28 [21ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:28 [23ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 05:16:28 [25ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 05:16:28 [28ms] [Info] [carb] [3037] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 05:16:28 [28ms] [Info] [carb] [3037] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:28 [28ms] [Info] [carb] [3037] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:28 [28ms] [Info] [carb] [3037] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:28 [28ms] [Info] [carb] [3037] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:28 [28ms] [Info] [carb] [3037] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:28 [29ms] [Info] [omni.server] [3037] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 05:16:28 [29ms] [Info] [omni.server] [3037] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 05:16:28 [29ms] [Info] [omni.server] [3037] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 05:16:28 [29ms] [Info] [omni.server] [3037] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 05:16:28 [29ms] [Info] [omni.server] [3037] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 05:16:28 [29ms] [Info] [omni.server] [3037] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 05:16:28 [29ms] [Info] [omni.server] [3037] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 05:16:28 [29ms] [Info] [omni.server] [3037] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 05:16:28 [29ms] [Info] [omni.server] [3037] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 05:16:28 [30ms] [Info] [omni.server] [3037] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 05:16:28 [30ms] [Info] [carb] [3037] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:28 [32ms] [Info] [carb] [3037] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:28 [32ms] [Info] [carb] [3037] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:28 [34ms] [Info] [carb] [3037] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:28 [35ms] [Info] [carb.launcher.plugin] [3037] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1e941c0 for './omni-lft-service/omni-lft-service'. 2023-02-19 05:16:28 [35ms] [Info] [carb.launcher.plugin] [3037] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3037. 2023-02-19 05:16:28 [35ms] [Info] [carb] [3037] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 05:16:28 [35ms] [Info] [carb] [3037] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 05:16:28 [35ms] [Info] [carb.launcher.plugin] [3054] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3055 from the parent 3037. 2023-02-19 05:16:28 [35ms] [Info] [carb.launcher.plugin] [3054] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3055. 2023-02-19 05:16:28 [37ms] [Info] [carb.launcher.plugin] [3054] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3055. 2023-02-19 05:16:28 [37ms] [Info] [carb.launcher.plugin] [3037] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3055 from the parent 3037. 2023-02-19 05:16:28 [37ms] [Info] [carb.launcher.plugin] [3056] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3055. 2023-02-19 05:16:28 [37ms] [Info] [carb.launcher.plugin] [3037] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3055 {process = 0x1e941c0} 2023-02-19 05:16:28 [37ms] [Info] [omni.core] [3037] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 05:16:28 [37ms] [Info] [carb.launcher.plugin] [3057] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3055. 2023-02-19 05:16:28 [37ms] [Info] [omni.core] [3037] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 05:16:28 [440ms] [Error] [omni.core] [3057] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 05:16:28 [446ms] [Error] [omni.core] [3057] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:16:28,915 - system_log - INFO - reconnecting ... 2023-02-19 05:16:28 [456ms] [Error] [omni.core] [3057] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:16:28,926 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 05:16:29 [0ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:29 [0ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:16:29 [0ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 05:16:29 [0ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:29 [0ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:29 [0ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:29 [0ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 05:16:29 [0ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:29 [0ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:29 [0ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:29 [1ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:29 [1ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 05:16:29 [1ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 05:16:29 [1ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 05:16:29 [1ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 05:16:29 [1ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 05:16:29 [1ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 05:16:29 [1ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 05:16:29 [1ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 05:16:29 [1ms] [Info] [omni.server] [3234] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 05:16:29 [2ms] [Info] [omni.server] [3234] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 05:16:29 [2ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:29 [2ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:29 [2ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:29 [3ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:29 [3ms] [Info] [carb.launcher.plugin] [3234] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1dca140 for './omni-lft-service/omni-lft-service'. 2023-02-19 05:16:29 [3ms] [Info] [carb.launcher.plugin] [3234] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3234. 2023-02-19 05:16:29 [3ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 05:16:29 [3ms] [Info] [carb] [3234] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 05:16:29 [3ms] [Info] [carb.launcher.plugin] [3246] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3247 from the parent 3234. 2023-02-19 05:16:29 [3ms] [Info] [carb.launcher.plugin] [3246] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3247. 2023-02-19 05:16:29 [4ms] [Info] [carb.launcher.plugin] [3246] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3247. 2023-02-19 05:16:29 [4ms] [Info] [carb.launcher.plugin] [3234] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3247 from the parent 3234. 2023-02-19 05:16:29 [4ms] [Info] [carb.launcher.plugin] [3234] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3247 {process = 0x1dca140} 2023-02-19 05:16:29 [4ms] [Info] [carb.launcher.plugin] [3248] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3247. 2023-02-19 05:16:29 [4ms] [Info] [omni.core] [3234] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 05:16:29 [4ms] [Info] [omni.core] [3234] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 05:16:29 [4ms] [Info] [carb.launcher.plugin] [3249] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3247. 2023-02-19 05:16:29 [272ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 05:16:29 [276ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:16:29,491 - system_log - INFO - reconnecting ... 2023-02-19 05:16:29 [285ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:16:29,500 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 05:16:30 [1,287ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 22:16:30,502 - system_log - INFO - reconnecting ... 2023-02-19 05:16:30 [1,289ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:16:30,504 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 05:16:31 [2,005ms] [Info] [omni.core] [3240] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 05:16:31 [2,005ms] [Info] [omni.core] [3240] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 05:16:31 [2,006ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:16:31 [2,006ms] [Info] [omni.core] [3240] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 05:16:31 [2,013ms] [Info] [omni.core] [3234] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 05:16:31 [2,013ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:16:31 [2,036ms] [Warning] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 05:16:31 [2,036ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 05:16:31 [2,045ms] [Warning] [carb] [3234] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-02-19 05:16:31 [2,045ms] [Info] [carb] [3234] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 7 records 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 7 records 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 7 records 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-02-19 05:16:31 [2,046ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-02-19 05:16:31 [2,047ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-02-19 05:16:31 [2,047ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-02-19 05:16:31 [2,047ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-02-19 05:16:31 [2,047ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-02-19 05:16:31 [2,047ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-02-19 05:16:31 [2,047ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-02-19 05:16:31 [2,047ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-02-19 05:16:31 [2,047ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-02-19 05:16:31 [2,047ms] [Info] [omni.meta-upgrade] [3234] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-02-19 05:16:31 [2,048ms] [Info] [omni.core] [3234] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-02-19 05:16:31 [2,055ms] [Info] [omni.core] [3234] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-02-19 05:16:31 [2,061ms] [Info] [omni.core] [3234] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-02-19 05:16:31 [2,061ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-02-19 05:16:31 [2,061ms] [Info] [omni.core] [3239] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-02-19 05:16:31 [2,062ms] [Info] [omni.core] [3238] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-02-19 05:16:31 [2,062ms] [Info] [carb.launcher.plugin] [3238] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7f7890003930 for './omni.delta-applier.app'. 2023-02-19 05:16:31 [2,062ms] [Info] [carb.launcher.plugin] [3238] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 3234. 2023-02-19 05:16:31 [2,063ms] [Info] [carb.launcher.plugin] [3246] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3334 from the parent 3234. 2023-02-19 05:16:31 [2,063ms] [Info] [carb.launcher.plugin] [3246] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3334. 2023-02-19 05:16:31 [2,066ms] [Info] [carb.launcher.plugin] [3246] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3334. 2023-02-19 05:16:31 [2,066ms] [Info] [carb.launcher.plugin] [3238] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3334 from the parent 3234. 2023-02-19 05:16:31 [2,066ms] [Info] [carb.launcher.plugin] [3238] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3334 {process = 0x7f7890003930} 2023-02-19 05:16:31 [2,066ms] [Info] [carb.launcher.plugin] [3335] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3334. 2023-02-19 05:16:31 [2,066ms] [Info] [carb.launcher.plugin] [3336] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3334. 2023-02-19 05:16:31 [2,291ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 22:16:31,506 - system_log - INFO - reconnecting ... 2023-02-19 05:16:31 [2,292ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:16:31,508 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 05:16:31 [2,373ms] [Info] [omni.core] [3238] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-02-19 05:16:32 [3,295ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 22:16:32,510 - system_log - INFO - reconnecting ... 2023-02-19 05:16:32 [3,297ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:16:32,512 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-19 05:16:33 [4,062ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-02-19 05:16:33 [4,068ms] [Info] [omni.server] [3234] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-02-19 05:16:33 [4,068ms] [Info] [omni.signalhandler] [3234] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-02-19 05:16:33 [4,068ms] [Info] [omni.signalhandler] [3234] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-02-19 05:16:33 [4,068ms] [Info] [omni.signalhandler] [3234] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-02-19 05:16:33 [4,299ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-02-18 22:16:33,514 - system_log - INFO - reconnecting ... 2023-02-19 05:16:33 [4,302ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:16:33,518 - system_log - INFO - connected 2023-02-19 05:16:33 [4,303ms] [Error] [omni.core] [3249] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-02-19 05:16:48 [0ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3392] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:48 [1ms] [Info] [omni.server] [3392] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 05:16:48 [1ms] [Info] [omni.server] [3392] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3392] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3392] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3392] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3392] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3392] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3392] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3392] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3392] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "" } } } 2023-02-19 05:16:48 [2ms] [Info] [carb] [3392] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:48 [2ms] [Info] [carb] [3392] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:48 [3ms] [Info] [carb] [3392] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:48 [4ms] [Info] [carb] [3392] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:48 [4ms] [Info] [carb.launcher.plugin] [3392] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x139c3b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 05:16:48 [4ms] [Info] [carb.launcher.plugin] [3392] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3392. 2023-02-19 05:16:48 [4ms] [Info] [carb] [3392] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 05:16:48 [4ms] [Info] [carb] [3392] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 05:16:48 [4ms] [Info] [carb.launcher.plugin] [3419] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3420 from the parent 3392. 2023-02-19 05:16:48 [4ms] [Info] [carb.launcher.plugin] [3419] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3420. 2023-02-19 05:16:48 [6ms] [Info] [carb.launcher.plugin] [3419] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3420. 2023-02-19 05:16:48 [6ms] [Info] [carb.launcher.plugin] [3392] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3420 from the parent 3392. 2023-02-19 05:16:48 [6ms] [Info] [carb.launcher.plugin] [3392] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3420 {process = 0x139c3b0} 2023-02-19 05:16:48 [6ms] [Info] [carb.launcher.plugin] [3424] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3420. 2023-02-19 05:16:48 [6ms] [Info] [omni.core] [3392] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 05:16:48 [6ms] [Info] [omni.core] [3392] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 05:16:48 [6ms] [Info] [carb.launcher.plugin] [3425] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3420. 2023-02-19 05:16:48 [0ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:48 [0ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:48 [1ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:48 [2ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 05:16:48 [2ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 05:16:48 [2ms] [Info] [carb] [3427] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 05:16:48 [2ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:48 [2ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:48 [2ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:48 [2ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:48 [2ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3427] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3427] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3427] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3427] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3427] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 05:16:48 [2ms] [Info] [omni.server] [3427] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 05:16:48 [3ms] [Info] [omni.server] [3427] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 05:16:48 [3ms] [Info] [omni.server] [3427] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 05:16:48 [3ms] [Info] [omni.server] [3427] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 05:16:48 [3ms] [Info] [omni.server] [3427] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 05:16:48 [3ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:48 [3ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:48 [4ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:48 [5ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:48 [5ms] [Info] [carb.launcher.plugin] [3427] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1e8c3b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 05:16:48 [5ms] [Info] [carb.launcher.plugin] [3427] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3427. 2023-02-19 05:16:48 [5ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 05:16:48 [5ms] [Info] [carb] [3427] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 05:16:48 [5ms] [Info] [carb.launcher.plugin] [3439] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3440 from the parent 3427. 2023-02-19 05:16:48 [5ms] [Info] [carb.launcher.plugin] [3439] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3440. 2023-02-19 05:16:48 [6ms] [Info] [carb.launcher.plugin] [3439] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3440. 2023-02-19 05:16:48 [6ms] [Info] [carb.launcher.plugin] [3427] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3440 from the parent 3427. 2023-02-19 05:16:48 [6ms] [Info] [carb.launcher.plugin] [3427] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3440 {process = 0x1e8c3b0} 2023-02-19 05:16:48 [6ms] [Info] [omni.core] [3427] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 05:16:48 [6ms] [Info] [carb.launcher.plugin] [3441] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3440. 2023-02-19 05:16:48 [6ms] [Info] [omni.core] [3427] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 05:16:48 [6ms] [Info] [carb.launcher.plugin] [3442] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3440. 2023-02-19 05:16:48 [264ms] [Error] [omni.core] [3442] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 05:16:48 [268ms] [Error] [omni.core] [3442] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:16:48,732 - system_log - INFO - reconnecting ... 2023-02-19 05:16:48 [278ms] [Error] [omni.core] [3442] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:16:48,743 - system_log - INFO - connected 2023-02-19 05:16:48 [278ms] [Error] [omni.core] [3442] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [3440] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 05:16:48 [312ms] [Info] [carb.launcher.plugin] [3441] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 05:16:48 [312ms] [Info] [carb.launcher.plugin] [3441] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3440, bytes = 0} 2023-02-19 05:16:48 [313ms] [Info] [carb.launcher.plugin] [3441] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 3440. 2023-02-19 05:16:48 [312ms] [Info] [carb.launcher.plugin] [3442] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 05:16:48 [313ms] [Info] [carb.launcher.plugin] [3442] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3440, bytes = 0} 2023-02-19 05:16:48 [313ms] [Info] [carb.launcher.plugin] [3442] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 3440. 2023-02-19 05:16:50 [2,006ms] [Info] [omni.core] [3432] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 05:16:50 [2,006ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:16:50 [2,006ms] [Info] [omni.core] [3432] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 05:16:50 [2,007ms] [Info] [omni.core] [3427] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 05:16:50 [2,007ms] [Info] [carb] [3427] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:16:50 [2,007ms] [Warning] [omni.keyvaluedatabase.leveldb] [3427] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 05:16:50 [2,008ms] [Info] [omni.core] [3432] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 05:16:50 [2,015ms] [Warning] [omni.keyvaluedatabase.leveldb] [3427] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 05:16:50 [2,015ms] [Fatal] [omni.core] [3427] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-02-19 05:16:59 [0ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3517] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3517] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3517] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3517] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3517] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3517] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3517] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3517] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3517] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3517] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 05:16:59 [2ms] [Info] [omni.server] [3517] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "" } } } 2023-02-19 05:16:59 [2ms] [Info] [carb] [3517] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:59 [2ms] [Info] [carb] [3517] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:59 [2ms] [Info] [carb] [3517] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:59 [2ms] [Info] [carb] [3517] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:59 [3ms] [Info] [carb.launcher.plugin] [3517] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x15363b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 05:16:59 [3ms] [Info] [carb.launcher.plugin] [3517] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3517. 2023-02-19 05:16:59 [3ms] [Info] [carb] [3517] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 05:16:59 [3ms] [Info] [carb] [3517] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 05:16:59 [3ms] [Info] [carb.launcher.plugin] [3538] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3539 from the parent 3517. 2023-02-19 05:16:59 [3ms] [Info] [carb.launcher.plugin] [3538] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3539. 2023-02-19 05:16:59 [4ms] [Info] [carb.launcher.plugin] [3538] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3539. 2023-02-19 05:16:59 [4ms] [Info] [carb.launcher.plugin] [3517] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3539 from the parent 3517. 2023-02-19 05:16:59 [4ms] [Info] [carb.launcher.plugin] [3517] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3539 {process = 0x15363b0} 2023-02-19 05:16:59 [4ms] [Info] [omni.core] [3517] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 05:16:59 [4ms] [Info] [omni.core] [3517] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 05:16:59 [4ms] [Info] [carb.launcher.plugin] [3543] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3539. 2023-02-19 05:16:59 [4ms] [Info] [carb.launcher.plugin] [3542] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3539. 2023-02-19 05:16:59 [0ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:59 [0ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:16:59 [1ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3551] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3551] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3551] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3551] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3551] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3551] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3551] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3551] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 05:16:59 [1ms] [Info] [omni.server] [3551] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 05:16:59 [2ms] [Info] [omni.server] [3551] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 05:16:59 [2ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:16:59 [2ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:16:59 [2ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:16:59 [3ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:16:59 [3ms] [Info] [carb.launcher.plugin] [3551] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x28663b0 for './omni-lft-service/omni-lft-service'. 2023-02-19 05:16:59 [3ms] [Info] [carb.launcher.plugin] [3551] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3551. 2023-02-19 05:16:59 [3ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 05:16:59 [3ms] [Info] [carb] [3551] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 05:16:59 [3ms] [Info] [carb.launcher.plugin] [3563] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3564 from the parent 3551. 2023-02-19 05:16:59 [3ms] [Info] [carb.launcher.plugin] [3563] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3564. 2023-02-19 05:16:59 [4ms] [Info] [carb.launcher.plugin] [3563] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3564. 2023-02-19 05:16:59 [4ms] [Info] [carb.launcher.plugin] [3551] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3564 from the parent 3551. 2023-02-19 05:16:59 [4ms] [Info] [carb.launcher.plugin] [3551] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3564 {process = 0x28663b0} 2023-02-19 05:16:59 [4ms] [Info] [omni.core] [3551] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 05:16:59 [4ms] [Info] [omni.core] [3551] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 05:16:59 [4ms] [Info] [carb.launcher.plugin] [3566] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3564. 2023-02-19 05:16:59 [4ms] [Info] [carb.launcher.plugin] [3565] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3564. 2023-02-19 05:17:00 [271ms] [Error] [omni.core] [3566] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 05:17:00 [275ms] [Error] [omni.core] [3566] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:17:00,199 - system_log - INFO - reconnecting ... 2023-02-19 05:17:00 [286ms] [Error] [omni.core] [3566] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:17:00,210 - system_log - INFO - connected 2023-02-19 05:17:00 [286ms] [Error] [omni.core] [3566] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [3564] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 05:17:00 [321ms] [Info] [carb.launcher.plugin] [3565] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 05:17:00 [321ms] [Info] [carb.launcher.plugin] [3565] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3564, bytes = 0} 2023-02-19 05:17:00 [321ms] [Info] [carb.launcher.plugin] [3565] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 3564. 2023-02-19 05:17:00 [321ms] [Info] [carb.launcher.plugin] [3566] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 05:17:00 [321ms] [Info] [carb.launcher.plugin] [3566] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3564, bytes = 0} 2023-02-19 05:17:00 [321ms] [Info] [carb.launcher.plugin] [3566] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 3564. 2023-02-19 05:17:01 [2,004ms] [Info] [omni.core] [3555] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 05:17:01 [2,004ms] [Info] [omni.core] [3555] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 05:17:01 [2,005ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:17:01 [2,006ms] [Info] [omni.core] [3551] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 05:17:01 [2,006ms] [Info] [carb] [3551] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:17:01 [2,007ms] [Warning] [omni.keyvaluedatabase.leveldb] [3551] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 05:17:01 [2,007ms] [Info] [omni.core] [3555] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 05:17:01 [2,012ms] [Warning] [omni.keyvaluedatabase.leveldb] [3551] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 05:17:01 [2,012ms] [Fatal] [omni.core] [3551] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-02-19 05:30:19 [0ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:30:19 [0ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:30:19 [0ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-02-19 05:30:19 [0ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:30:19 [1ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:30:19 [1ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:30:19 [1ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-02-19 05:30:19 [1ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:30:19 [1ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:30:19 [1ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:30:19 [2ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:30:19 [2ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:30:19 [2ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:30:19 [2ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-02-19 05:30:19 [2ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-02-19 05:30:19 [2ms] [Info] [carb] [8711] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-02-19 05:30:19 [3ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-02-19 05:30:19 [3ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-02-19 05:30:19 [3ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-02-19 05:30:19 [3ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-02-19 05:30:19 [3ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-02-19 05:30:19 [3ms] [Info] [omni.server] [8711] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-02-19 05:30:19 [3ms] [Info] [omni.server] [8711] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-02-19 05:30:19 [3ms] [Info] [omni.server] [8711] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-02-19 05:30:19 [3ms] [Info] [omni.server] [8711] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-02-19 05:30:19 [3ms] [Info] [omni.server] [8711] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/elena/.local/share/ov/data/server 2023-02-19 05:30:19 [3ms] [Info] [omni.server] [8711] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/elena/.cache/ov/server 2023-02-19 05:30:19 [3ms] [Info] [omni.server] [8711] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-02-19 05:30:19 [3ms] [Info] [omni.server] [8711] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/elena/.nvidia-omniverse/logs/Nucleus 2023-02-19 05:30:19 [4ms] [Info] [omni.server] [8711] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-02-19 05:30:19 [4ms] [Info] [omni.server] [8711] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "log": { "fileLogLevel": "verbose", "detail": false, "includeChannel": true, "includeProcessId": false, "includeSource": true, "processGroupId": 0, "debugConsoleLevel": "verbose", "enableColorOutput": true, "enableDebugConsoleOutput": true, "fileFlushLevel": "verbose", "sources": { "omni.server.API_CLIENT": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false } }, "outputStreamLevel": "verbose", "flushStandardStreamOutput": false, "includeThreadId": true, "fullDetail": false, "includeLineNumber": true, "enabled": true, "enableStandardStreamOutput": true, "includeFilename": true, "fileAppend": true, "outputStream": "stderr", "setElapsedTimeUnits": "milliseconds", "includeTimeStamp": true, "includeFunctionName": true, "file": "/home/elena/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "level": "Info", "async": false }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "pluginSearchPaths": [ "." ], "crashreporter": { "retryCount": 10, "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "data": { "buildVersion": "114.4.0", "buildMajor": "114", "buildNumber": "21980", "buildMinor": "4", "buildHash": "84b6cd59", "buildBranch": "master", "buildCi": "gitlab", "buildPatch": "0", "buildMr": "0" }, "version": "114.4+master.gitlab.21980.84b6cd59", "url": "https://services.nvidia.com/submit", "dumpFlags": "", "log": "true", "preserveDump": false, "dumpDir": ".", "product": "OmniverseNucleus", "alwaysUpload": false, "debuggerAttachTimeoutMs": 0, "enabled": "true", "skipOldDumpUpload": false, "uploadTimeoutMs": 7200000 }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "omni.server.app": {}, "privacy": { "performance": "false" }, "plugins": { "omni.websocketslib": { "tracing": false, "defaults": { "http_max_header_length": "61440", "tcp_timeout_seconds": 10, "tcp_keepalive_time": 10, "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2 } }, "omni.meta-upgrade": { "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "exit_on_any_failure": false, "interactive": false, "report_progress_every": 1000, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } }, "remove_deduplicated_content": true, "convertors": { "meta": { "dry_run_since_version": -1 } } }, "omni.messaging-redis": { "host": "kube-msk-redis.nvidia.com", "port": "6379", "password": "yBGfkZ0iRT", "max_reconnects": 5, "reconnect_interval_ms": 1000, "timeout_ms": 500 }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "debugWaitingTasks": false, "stuckCheckSeconds": 1, "resetFiberStack": true, "fiberCount": 4095, "threadCount": "4" }, "omni.metrics-prometheus": { "model": "pull", "pull": { "host": "127.0.0.1", "port": 3010, "endpoint": "/" }, "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "predefined_labels": { "omni_instance": "" } }, "omni.server": { "delta_content_hang_timeout": 2.0, "default_branch_name": "default", "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "meta.1.1" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "disable_passwords": false, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "file_system_table": true, "multipart_chunk_size_mb": "5", "mount_management_requires_admin": true, "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "record_integrity_check": false, "delta_manager": { "delta_apply_throttle_ms": 500, "delta_apply_limit_mb": 128, "bw_estimate_bps": 256, "num_retries_until_ignored": 10, "ignorelist_retry_delay_ms": 5000, "reconnect_timeout_s": 20, "delta_apply_timeout_ms": 5000, "ping_interval_ms": 1000, "ping_interval_on_startup_ms": 100, "ping_timeout_ms": 1000 }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "mounts": { "default_mounts": [ { "path": "/NVIDIA", "resolver": "omniverse_resolver_s3", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "override": true } ], "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {}, "logging_enabled": "false" }, "password_recovery": true, "crash_record_recovery": true, "max_path_piece_length": "255", "token_expire_time": 86400, "delta_content_ping_interval": 0.0, "content_flush_enabled": true, "master_password": "omniverse", "lft_content_expiration_time": "600", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "elem_size_max": 1048576, "number_elems_low": 100000, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "total_size_high": 6442450944, "temporal_kind": "omniverse_resolver_memory", "total_size_low": 4294967296, "temporal_options": {}, "persistent_kind": "omniverse_resolver_file" }, "logging_enabled": "true" }, "non_utf8_paths_allowed": "false", "allowed_users": {}, "omniobjects": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "omniobjects.1.0" }, "max_objects_in_cache": 1024, "lft_read_timeout_seconds": 3600, "max_content_mem_cache_size": 104857600, "lft_multi_content_threshold": 0, "unpack_object_buffer_size": 33554432, "ext_value_min_size": 262144, "mem_vs_file_threshold": 262144, "content_mem_limit": 268435456, "max_content_file_cache_size": 1048576000 }, "default_s3_proxy_cache": "", "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "endpoints": { "api": [ { "interface": "127.0.0.1", "port": "3009", "secure": false, "lft_address": "http://*:3030" } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ] }, "store_crash_repro": true, "dirs": { "content_dir": "/home/elena/.local/share/ov/data/server", "meta_dir": "/home/elena/.local/share/ov/data/server", "omniobjects_dir": "/home/elena/.local/share/ov/data/server", "metric_dir": "/home/elena/.local/share/ov/data/server", "record_dir": "", "usergroups_dir": "/home/elena/.local/share/ov/data/server", "log_dir": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "list_batch_size_limit": 100, "run": { "versioning": "true" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "peer_messaging": { "enable": "false", "exit_on_connection_failure": "false", "channel": "omni_server_messages", "application_name": "Nucleus API", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "resolver_options": { "async": true, "root": "" }, "logging_enabled": "true" }, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "api_acls": { "list": "r", "unlock": "w", "rename": "a", "lock": "w", "delete": "a", "get_acl_resolved": "r", "get_checkpoints": "r", "checkpoint_version": "w", "unmount": "w", "create_asset": "w", "create_directory": "w", "mount": "w", "update": "w", "create_object": "w", "copy": "w", "get_acl_v2": "r", "set_acl_v2": "a", "set_path_options": "w", "read": "r" }, "allowed_users_legacy": {}, "test_token_prefix": "", "metric_period": 10.0, "content_type": "", "auto_create_users": true, "delta_content_bw_estimate": 0.5, "discovery": { "enabled": "true", "host": "localhost", "token": "test", "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }" }, "content_flush_time_sec": 10, "removed_content_pending_time_sec": 5, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "metacache_flush_count": 1000, "notifications": {}, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "delete_on_startup": "false", "root": "", "subdir": "usergroups.1.0" } }, "file_and_username_filter_exclude_regex": "", "api_logging": { "json_outputs": {}, "omit_event_types": {}, "event_buffer_max_size_bytes": 1048576, "event_buffer_min_size_bytes": 1024, "enable": "false", "event_buffer_try_shrink_interval_sec": 10, "enable_legacy_system": "1", "source_base_name": "api-server", "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "source_instance_name": "", "tsv_outputs": [ { "omit_columns": {}, "files": [ "eventlog.tsv" ], "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "print_header_on_new_field": true, "omit_unknown_columns": true } ] } }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] } }, "APILOGGING": 1, "LFT": { "INTERFACE": "127.0.0.1" }, "LOG": { "DIR": "/home/elena/.nvidia-omniverse/logs/Nucleus" }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } } } 2023-02-19 05:30:19 [4ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-02-19 05:30:19 [5ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-02-19 05:30:19 [5ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-02-19 05:30:19 [6ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-02-19 05:30:19 [6ms] [Info] [carb.launcher.plugin] [8711] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xed61c0 for './omni-lft-service/omni-lft-service'. 2023-02-19 05:30:19 [6ms] [Info] [carb.launcher.plugin] [8711] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 8711. 2023-02-19 05:30:19 [6ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-02-19 05:30:19 [6ms] [Info] [carb] [8711] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-02-19 05:30:19 [7ms] [Info] [carb.launcher.plugin] [8723] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 8724 from the parent 8711. 2023-02-19 05:30:19 [7ms] [Info] [carb.launcher.plugin] [8723] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 8724. 2023-02-19 05:30:19 [8ms] [Info] [carb.launcher.plugin] [8723] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 8724. 2023-02-19 05:30:19 [8ms] [Info] [carb.launcher.plugin] [8711] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 8724 from the parent 8711. 2023-02-19 05:30:19 [8ms] [Info] [carb.launcher.plugin] [8711] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 8724 {process = 0xed61c0} 2023-02-19 05:30:19 [8ms] [Info] [omni.core] [8711] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/elena/.local/share/ov/pkg/nucleus-workstation-2022.4.2/Nucleus/./omni-lft-service/omni-lft-service) 2023-02-19 05:30:19 [8ms] [Info] [omni.core] [8711] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-02-19 05:30:19 [8ms] [Info] [carb.launcher.plugin] [8725] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 8724. 2023-02-19 05:30:19 [8ms] [Info] [carb.launcher.plugin] [8726] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 8724. 2023-02-19 05:30:19 [253ms] [Error] [omni.core] [8726] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/elena/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-02-19 05:30:19 [257ms] [Error] [omni.core] [8726] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:30:19,610 - system_log - INFO - reconnecting ... 2023-02-19 05:30:19 [267ms] [Error] [omni.core] [8726] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-02-18 22:30:19,620 - system_log - INFO - connected 2023-02-19 05:30:19 [268ms] [Error] [omni.core] [8726] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [8724] Failed to execute script 'run_server' due to unhandled exception! 2023-02-19 05:30:19 [293ms] [Info] [carb.launcher.plugin] [8725] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 05:30:19 [293ms] [Info] [carb.launcher.plugin] [8726] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-02-19 05:30:19 [293ms] [Info] [carb.launcher.plugin] [8725] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 8724, bytes = 0} 2023-02-19 05:30:19 [293ms] [Info] [carb.launcher.plugin] [8726] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 8724, bytes = 0} 2023-02-19 05:30:19 [293ms] [Info] [carb.launcher.plugin] [8725] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 8724. 2023-02-19 05:30:19 [293ms] [Info] [carb.launcher.plugin] [8726] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 8724. 2023-02-19 05:30:21 [2,009ms] [Info] [omni.core] [8715] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-02-19 05:30:21 [2,009ms] [Info] [omni.core] [8715] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-02-19 05:30:21 [2,009ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-02-19 05:30:21 [2,011ms] [Info] [omni.core] [8711] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-02-19 05:30:21 [2,012ms] [Info] [carb] [8711] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-02-19 05:30:21 [2,012ms] [Warning] [omni.keyvaluedatabase.leveldb] [8711] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-02-19 05:30:21 [2,013ms] [Info] [omni.core] [8715] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-02-19 05:30:21 [2,017ms] [Warning] [omni.keyvaluedatabase.leveldb] [8711] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-02-19 05:30:21 [2,017ms] [Fatal] [omni.core] [8711] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta'