2023-01-03 07:59:46 [0ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2685] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2685] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2685] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2685] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2685] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2685] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2685] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2685] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2685] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2685] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2685] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-03 07:59:46 [3ms] [Info] [carb] [2685] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 07:59:46 [6ms] [Info] [carb] [2685] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 07:59:46 [6ms] [Info] [carb] [2685] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 07:59:46 [7ms] [Info] [carb] [2685] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 07:59:46 [7ms] [Info] [carb.launcher.plugin] [2685] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x15a1700 for './omni-lft-service/omni-lft-service'. 2023-01-03 07:59:46 [7ms] [Info] [carb.launcher.plugin] [2685] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 2685. 2023-01-03 07:59:46 [7ms] [Info] [carb] [2685] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-03 07:59:46 [7ms] [Info] [carb] [2685] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-03 07:59:46 [7ms] [Info] [carb.launcher.plugin] [2729] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 2730 from the parent 2685. 2023-01-03 07:59:46 [7ms] [Info] [carb.launcher.plugin] [2729] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 2730. 2023-01-03 07:59:46 [8ms] [Info] [carb.launcher.plugin] [2729] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 2730. 2023-01-03 07:59:46 [8ms] [Info] [carb.launcher.plugin] [2685] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 2730 from the parent 2685. 2023-01-03 07:59:46 [8ms] [Info] [carb.launcher.plugin] [2685] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 2730 {process = 0x15a1700} 2023-01-03 07:59:46 [8ms] [Info] [omni.core] [2685] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-03 07:59:46 [8ms] [Info] [carb.launcher.plugin] [2731] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 2730. 2023-01-03 07:59:46 [8ms] [Info] [omni.core] [2685] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-03 07:59:46 [0ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 07:59:46 [0ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 07:59:46 [1ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-03 07:59:46 [2ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 07:59:46 [2ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 07:59:46 [2ms] [Info] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 07:59:46 [2ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 07:59:46 [2ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 07:59:46 [2ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 07:59:46 [2ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 07:59:46 [2ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 07:59:46 [2ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-03 07:59:46 [2ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-03 07:59:46 [2ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-03 07:59:46 [2ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-03 07:59:46 [2ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-03 07:59:46 [2ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-03 07:59:46 [2ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-03 07:59:46 [2ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2733] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-03 07:59:46 [3ms] [Info] [omni.server] [2733] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-03 07:59:46 [3ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 07:59:46 [4ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 07:59:46 [4ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 07:59:46 [5ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 07:59:46 [5ms] [Info] [carb.launcher.plugin] [2733] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xaa16f0 for './omni-lft-service/omni-lft-service'. 2023-01-03 07:59:46 [5ms] [Info] [carb.launcher.plugin] [2733] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 2733. 2023-01-03 07:59:46 [5ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-03 07:59:46 [5ms] [Info] [carb] [2733] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-03 07:59:46 [5ms] [Info] [carb.launcher.plugin] [2798] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 2799 from the parent 2733. 2023-01-03 07:59:46 [5ms] [Info] [carb.launcher.plugin] [2798] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 2799. 2023-01-03 07:59:46 [6ms] [Info] [carb.launcher.plugin] [2798] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 2799. 2023-01-03 07:59:46 [6ms] [Info] [carb.launcher.plugin] [2733] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 2799 from the parent 2733. 2023-01-03 07:59:46 [6ms] [Info] [carb.launcher.plugin] [2733] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 2799 {process = 0xaa16f0} 2023-01-03 07:59:46 [6ms] [Info] [omni.core] [2733] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-03 07:59:46 [6ms] [Info] [omni.core] [2733] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-03 07:59:46 [6ms] [Info] [carb.launcher.plugin] [2800] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 2799. 2023-01-03 07:59:46 [6ms] [Info] [carb.launcher.plugin] [2801] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 2799. 2023-01-03 07:59:46 [269ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-03 07:59:46 [272ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 16:59:46,997 - system_log - INFO - reconnecting ... 2023-01-03 07:59:47 [280ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 16:59:47,005 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 07:59:48 [1,282ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 16:59:48,006 - system_log - INFO - reconnecting ... 2023-01-03 07:59:48 [1,282ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 16:59:48,007 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 07:59:48 [2,006ms] [Info] [omni.core] [2737] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-03 07:59:48 [2,006ms] [Info] [omni.core] [2737] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-03 07:59:48 [2,006ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 07:59:48 [2,007ms] [Info] [omni.core] [2733] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-03 07:59:48 [2,007ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 07:59:48 [2,007ms] [Info] [omni.core] [2737] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-03 07:59:48 [2,072ms] [Warning] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-03 07:59:48 [2,072ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-03 07:59:48 [2,096ms] [Warning] [carb] [2733] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-03 07:59:48 [2,096ms] [Info] [carb] [2733] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-03 07:59:48 [2,096ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-01-03 07:59:48 [2,105ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-01-03 07:59:48 [2,105ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 17536 records 2023-01-03 07:59:48 [2,111ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1001 / 17536 records 2023-01-03 07:59:48 [2,114ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 2001 / 17536 records 2023-01-03 07:59:48 [2,117ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 3001 / 17536 records 2023-01-03 07:59:48 [2,120ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 4001 / 17536 records 2023-01-03 07:59:48 [2,122ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 5001 / 17536 records 2023-01-03 07:59:48 [2,125ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 6001 / 17536 records 2023-01-03 07:59:48 [2,126ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 7001 / 17536 records 2023-01-03 07:59:48 [2,127ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 8001 / 17536 records 2023-01-03 07:59:48 [2,127ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 9001 / 17536 records 2023-01-03 07:59:48 [2,127ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 10001 / 17536 records 2023-01-03 07:59:48 [2,129ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 11001 / 17536 records 2023-01-03 07:59:48 [2,129ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 12001 / 17536 records 2023-01-03 07:59:48 [2,129ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 13001 / 17536 records 2023-01-03 07:59:48 [2,129ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 14001 / 17536 records 2023-01-03 07:59:48 [2,129ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 15001 / 17536 records 2023-01-03 07:59:48 [2,131ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 16001 / 17536 records 2023-01-03 07:59:48 [2,132ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 17001 / 17536 records 2023-01-03 07:59:48 [2,132ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-01-03 07:59:48 [2,134ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 4242 records 2023-01-03 07:59:48 [2,136ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1001 / 4242 records 2023-01-03 07:59:48 [2,138ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 2001 / 4242 records 2023-01-03 07:59:48 [2,139ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 3001 / 4242 records 2023-01-03 07:59:48 [2,141ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 4001 / 4242 records 2023-01-03 07:59:48 [2,142ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-01-03 07:59:48 [2,148ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-01-03 07:59:48 [2,148ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-01-03 07:59:48 [2,148ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 17536 records 2023-01-03 07:59:48 [2,148ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1001 / 17536 records 2023-01-03 07:59:48 [2,151ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 2001 / 17536 records 2023-01-03 07:59:48 [2,155ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 3001 / 17536 records 2023-01-03 07:59:48 [2,158ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 4001 / 17536 records 2023-01-03 07:59:48 [2,162ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 5001 / 17536 records 2023-01-03 07:59:48 [2,166ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 6001 / 17536 records 2023-01-03 07:59:48 [2,172ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 7001 / 17536 records 2023-01-03 07:59:48 [2,179ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 8001 / 17536 records 2023-01-03 07:59:48 [2,187ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 9001 / 17536 records 2023-01-03 07:59:48 [2,194ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 10001 / 17536 records 2023-01-03 07:59:48 [2,201ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 11001 / 17536 records 2023-01-03 07:59:48 [2,208ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 12001 / 17536 records 2023-01-03 07:59:48 [2,215ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 13001 / 17536 records 2023-01-03 07:59:48 [2,222ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 14001 / 17536 records 2023-01-03 07:59:48 [2,229ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 15001 / 17536 records 2023-01-03 07:59:48 [2,234ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 16001 / 17536 records 2023-01-03 07:59:48 [2,240ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 17001 / 17536 records 2023-01-03 07:59:48 [2,244ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-01-03 07:59:48 [2,244ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-01-03 07:59:48 [2,244ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1 / 7697 records 2023-01-03 07:59:48 [2,244ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1001 / 7697 records 2023-01-03 07:59:48 [2,245ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 2001 / 7697 records 2023-01-03 07:59:48 [2,246ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 3001 / 7697 records 2023-01-03 07:59:48 [2,247ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 4001 / 7697 records 2023-01-03 07:59:48 [2,248ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 5001 / 7697 records 2023-01-03 07:59:48 [2,249ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 6001 / 7697 records 2023-01-03 07:59:48 [2,250ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 7001 / 7697 records 2023-01-03 07:59:48 [2,250ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-01-03 07:59:48 [2,250ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-01-03 07:59:48 [2,259ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-01-03 07:59:48 [2,259ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-01-03 07:59:48 [2,259ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-03 07:59:48 [2,259ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-03 07:59:48 [2,259ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-03 07:59:48 [2,259ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-03 07:59:48 [2,259ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-03 07:59:48 [2,259ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-03 07:59:48 [2,259ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-03 07:59:48 [2,260ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-03 07:59:48 [2,260ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-03 07:59:48 [2,260ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-03 07:59:48 [2,261ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-03 07:59:48 [2,261ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-03 07:59:48 [2,261ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-03 07:59:48 [2,262ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-03 07:59:48 [2,262ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-03 07:59:48 [2,262ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-03 07:59:48 [2,263ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-03 07:59:48 [2,270ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-03 07:59:48 [2,276ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-03 07:59:49 [2,282ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-03 07:59:49 [2,284ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 16:59:49,008 - system_log - INFO - reconnecting ... 2023-01-03 07:59:49 [2,284ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 16:59:49,009 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 07:59:49 [2,289ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-03 07:59:49 [2,296ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-03 07:59:49 [2,302ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-03 07:59:49 [2,310ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-03 07:59:49 [2,319ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-03 07:59:49 [2,328ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-03 07:59:49 [2,336ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-03 07:59:49 [2,343ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-03 07:59:49 [2,352ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-03 07:59:49 [2,361ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-03 07:59:49 [2,369ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-03 07:59:49 [2,378ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-03 07:59:49 [2,387ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-03 07:59:49 [2,394ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-03 07:59:49 [2,397ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-01-03 07:59:49 [2,424ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-01-03 07:59:49 [2,424ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-01-03 07:59:49 [2,430ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-01-03 07:59:49 [2,431ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-01-03 07:59:49 [2,431ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-01-03 07:59:49 [2,431ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-01-03 07:59:49 [2,431ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-01-03 07:59:49 [2,431ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-01-03 07:59:49 [2,431ms] [Info] [omni.meta-upgrade] [2733] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-01-03 07:59:49 [2,445ms] [Info] [omni.core] [2733] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-01-03 07:59:49 [2,445ms] [Info] [omni.core] [2733] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-01-03 07:59:49 [2,447ms] [Info] [omni.core] [2733] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-01-03 07:59:49 [2,447ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-01-03 07:59:49 [2,447ms] [Info] [omni.core] [2738] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-01-03 07:59:49 [2,448ms] [Info] [omni.core] [2737] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-01-03 07:59:49 [2,448ms] [Info] [carb.launcher.plugin] [2737] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7fdfe40064f0 for './omni.delta-applier.app'. 2023-01-03 07:59:49 [2,448ms] [Info] [carb.launcher.plugin] [2737] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 2733. 2023-01-03 07:59:49 [2,449ms] [Info] [carb.launcher.plugin] [2798] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 2870 from the parent 2733. 2023-01-03 07:59:49 [2,449ms] [Info] [carb.launcher.plugin] [2798] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 2870. 2023-01-03 07:59:49 [2,450ms] [Info] [carb.launcher.plugin] [2798] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 2870. 2023-01-03 07:59:49 [2,450ms] [Info] [carb.launcher.plugin] [2737] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 2870 from the parent 2733. 2023-01-03 07:59:49 [2,450ms] [Info] [carb.launcher.plugin] [2737] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 2870 {process = 0x7fdfe40064f0} 2023-01-03 07:59:49 [2,450ms] [Info] [carb.launcher.plugin] [2871] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 2870. 2023-01-03 07:59:49 [2,450ms] [Info] [carb.launcher.plugin] [2872] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 2870. 2023-01-03 07:59:49 [2,651ms] [Info] [omni.core] [2739] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-01-03 07:59:50 [3,285ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 16:59:50,010 - system_log - INFO - reconnecting ... 2023-01-03 07:59:50 [3,286ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 16:59:50,011 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 07:59:51 [4,287ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 16:59:51,012 - system_log - INFO - reconnecting ... 2023-01-03 07:59:51 [4,288ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 16:59:51,012 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 07:59:51 [4,447ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-01-03 07:59:51 [4,449ms] [Error] [carb] [2733] ../../../source/framework/FileSystem.cpp:move():964: File move failed: '/home/nw/.nvidia-omniverse/logs/Nucleus/Access.log' -> '/home/nw/.nvidia-omniverse/logs/Nucleus/Access_OLD.log' (No such file or directory) 2023-01-03 07:59:51 [4,449ms] [Error] [omni.server] [2733] ../../../source/plugins/omni.server/apilogging/deprecated/ApiLogging.cpp:resetApiLogFile():210: Failed to rename access log file '/home/nw/.nvidia-omniverse/logs/Nucleus/Access.log' to '/home/nw/.nvidia-omniverse/logs/Nucleus/Access_OLD.log' 2023-01-03 07:59:51 [4,449ms] [Info] [omni.server] [2733] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-01-03 07:59:51 [4,449ms] [Info] [omni.signalhandler] [2733] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-01-03 07:59:51 [4,449ms] [Info] [omni.signalhandler] [2733] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-01-03 07:59:51 [4,449ms] [Info] [omni.signalhandler] [2733] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-01-03 07:59:52 [5,289ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 16:59:52,014 - system_log - INFO - reconnecting ... 2023-01-03 07:59:52 [5,291ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 16:59:52,015 - system_log - INFO - connected 2023-01-03 07:59:52 [5,291ms] [Error] [omni.core] [2801] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-01-03 08:28:32 [0ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-03 08:28:32 [0ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 08:28:32 [1ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 08:28:32 [1ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 08:28:32 [2ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 08:28:32 [2ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-03 08:28:32 [2ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 08:28:32 [3ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-03 08:28:32 [3ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 08:28:32 [3ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 08:28:32 [4ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-03 08:28:32 [4ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 08:28:32 [5ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-03 08:28:32 [5ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 08:28:32 [5ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 08:28:32 [5ms] [Info] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 08:28:32 [5ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 08:28:32 [5ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 08:28:32 [6ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 08:28:32 [6ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 08:28:32 [6ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 08:28:32 [24ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-03 08:28:32 [24ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-03 08:28:32 [24ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-03 08:28:32 [24ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-03 08:28:32 [24ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-03 08:28:32 [24ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-03 08:28:32 [24ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-03 08:28:32 [24ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-03 08:28:32 [25ms] [Info] [omni.server] [6291] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-03 08:28:32 [25ms] [Info] [omni.server] [6291] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "0.0.0.0" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "0.0.0.0" } }, "S3": { "RESOLVER": { "CACHE": "" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "0.0.0.0", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-03 08:28:32 [25ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 08:28:32 [40ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 08:28:32 [41ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 08:28:32 [41ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 08:28:32 [42ms] [Info] [carb.launcher.plugin] [6291] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xbc6a40 for './omni-lft-service/omni-lft-service'. 2023-01-03 08:28:32 [42ms] [Info] [carb.launcher.plugin] [6291] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 6291. 2023-01-03 08:28:32 [42ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-03 08:28:32 [42ms] [Info] [carb] [6291] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-03 08:28:32 [54ms] [Info] [carb.launcher.plugin] [6306] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 6307 from the parent 6291. 2023-01-03 08:28:32 [54ms] [Info] [carb.launcher.plugin] [6306] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 6307. 2023-01-03 08:28:32 [55ms] [Info] [carb.launcher.plugin] [6306] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 6307. 2023-01-03 08:28:32 [55ms] [Info] [carb.launcher.plugin] [6291] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 6307 from the parent 6291. 2023-01-03 08:28:32 [55ms] [Info] [carb.launcher.plugin] [6291] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 6307 {process = 0xbc6a40} 2023-01-03 08:28:32 [55ms] [Info] [omni.core] [6291] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-03 08:28:32 [55ms] [Info] [omni.core] [6291] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-03 08:28:32 [64ms] [Info] [carb.launcher.plugin] [6308] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 6307. 2023-01-03 08:28:32 [72ms] [Info] [carb.launcher.plugin] [6309] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 6307. 2023-01-03 08:28:32 [579ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-03 08:28:32 [586ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:28:32,643 - system_log - INFO - reconnecting ... 2023-01-03 08:28:32 [601ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:28:32,657 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 08:28:33 [1,603ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 17:28:33,659 - system_log - INFO - reconnecting ... 2023-01-03 08:28:33 [1,604ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:28:33,660 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 08:28:34 [2,055ms] [Info] [omni.core] [6296] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-03 08:28:34 [2,055ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 08:28:34 [2,055ms] [Info] [omni.core] [6296] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-03 08:28:34 [2,059ms] [Info] [omni.core] [6291] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-03 08:28:34 [2,059ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 08:28:34 [2,059ms] [Info] [omni.core] [6296] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-03 08:28:34 [2,135ms] [Warning] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-03 08:28:34 [2,135ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-03 08:28:34 [2,163ms] [Warning] [carb] [6291] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-03 08:28:34 [2,163ms] [Info] [carb] [6291] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-03 08:28:34 [2,163ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-01-03 08:28:34 [2,180ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-01-03 08:28:34 [2,180ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 17536 records 2023-01-03 08:28:34 [2,190ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1001 / 17536 records 2023-01-03 08:28:34 [2,196ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 2001 / 17536 records 2023-01-03 08:28:34 [2,201ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 3001 / 17536 records 2023-01-03 08:28:34 [2,206ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 4001 / 17536 records 2023-01-03 08:28:34 [2,211ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 5001 / 17536 records 2023-01-03 08:28:34 [2,216ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 6001 / 17536 records 2023-01-03 08:28:34 [2,219ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 7001 / 17536 records 2023-01-03 08:28:34 [2,220ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 8001 / 17536 records 2023-01-03 08:28:34 [2,221ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 9001 / 17536 records 2023-01-03 08:28:34 [2,221ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 10001 / 17536 records 2023-01-03 08:28:34 [2,223ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 11001 / 17536 records 2023-01-03 08:28:34 [2,223ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 12001 / 17536 records 2023-01-03 08:28:34 [2,223ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 13001 / 17536 records 2023-01-03 08:28:34 [2,224ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 14001 / 17536 records 2023-01-03 08:28:34 [2,224ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 15001 / 17536 records 2023-01-03 08:28:34 [2,226ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 16001 / 17536 records 2023-01-03 08:28:34 [2,229ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 17001 / 17536 records 2023-01-03 08:28:34 [2,229ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-01-03 08:28:34 [2,233ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 4242 records 2023-01-03 08:28:34 [2,236ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1001 / 4242 records 2023-01-03 08:28:34 [2,239ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 2001 / 4242 records 2023-01-03 08:28:34 [2,243ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 3001 / 4242 records 2023-01-03 08:28:34 [2,246ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 4001 / 4242 records 2023-01-03 08:28:34 [2,247ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-01-03 08:28:34 [2,263ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-01-03 08:28:34 [2,263ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-01-03 08:28:34 [2,263ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 17536 records 2023-01-03 08:28:34 [2,263ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1001 / 17536 records 2023-01-03 08:28:34 [2,268ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 2001 / 17536 records 2023-01-03 08:28:34 [2,275ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 3001 / 17536 records 2023-01-03 08:28:34 [2,282ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 4001 / 17536 records 2023-01-03 08:28:34 [2,288ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 5001 / 17536 records 2023-01-03 08:28:34 [2,295ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 6001 / 17536 records 2023-01-03 08:28:34 [2,306ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 7001 / 17536 records 2023-01-03 08:28:34 [2,320ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 8001 / 17536 records 2023-01-03 08:28:34 [2,332ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 9001 / 17536 records 2023-01-03 08:28:34 [2,344ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 10001 / 17536 records 2023-01-03 08:28:34 [2,357ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 11001 / 17536 records 2023-01-03 08:28:34 [2,369ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 12001 / 17536 records 2023-01-03 08:28:34 [2,381ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 13001 / 17536 records 2023-01-03 08:28:34 [2,393ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 14001 / 17536 records 2023-01-03 08:28:34 [2,405ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 15001 / 17536 records 2023-01-03 08:28:34 [2,416ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 16001 / 17536 records 2023-01-03 08:28:34 [2,427ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 17001 / 17536 records 2023-01-03 08:28:34 [2,434ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-01-03 08:28:34 [2,434ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-01-03 08:28:34 [2,434ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1 / 7697 records 2023-01-03 08:28:34 [2,436ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1001 / 7697 records 2023-01-03 08:28:34 [2,438ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 2001 / 7697 records 2023-01-03 08:28:34 [2,440ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 3001 / 7697 records 2023-01-03 08:28:34 [2,442ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 4001 / 7697 records 2023-01-03 08:28:34 [2,443ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 5001 / 7697 records 2023-01-03 08:28:34 [2,445ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 6001 / 7697 records 2023-01-03 08:28:34 [2,447ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 7001 / 7697 records 2023-01-03 08:28:34 [2,448ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-01-03 08:28:34 [2,448ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-01-03 08:28:34 [2,469ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-01-03 08:28:34 [2,469ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-01-03 08:28:34 [2,469ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-03 08:28:34 [2,469ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-03 08:28:34 [2,469ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-03 08:28:34 [2,470ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-03 08:28:34 [2,470ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-03 08:28:34 [2,470ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-03 08:28:34 [2,471ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-03 08:28:34 [2,471ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-03 08:28:34 [2,471ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-03 08:28:34 [2,471ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-03 08:28:34 [2,475ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-03 08:28:34 [2,475ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-03 08:28:34 [2,475ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-03 08:28:34 [2,475ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-03 08:28:34 [2,475ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-03 08:28:34 [2,476ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-03 08:28:34 [2,478ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-03 08:28:34 [2,492ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-03 08:28:34 [2,505ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-03 08:28:34 [2,519ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-03 08:28:34 [2,533ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-03 08:28:34 [2,546ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-03 08:28:34 [2,559ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-03 08:28:34 [2,575ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-03 08:28:34 [2,593ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-03 08:28:34 [2,604ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-03 08:28:34 [2,605ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 17:28:34,661 - system_log - INFO - reconnecting ... 2023-01-03 08:28:34 [2,606ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:28:34,662 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 08:28:34 [2,615ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-03 08:28:34 [2,624ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-03 08:28:34 [2,637ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-03 08:28:34 [2,649ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-03 08:28:34 [2,661ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-03 08:28:34 [2,673ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-03 08:28:34 [2,686ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-03 08:28:34 [2,695ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-03 08:28:34 [2,699ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-01-03 08:28:34 [2,740ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-01-03 08:28:34 [2,740ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-01-03 08:28:34 [2,752ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-01-03 08:28:34 [2,753ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-01-03 08:28:34 [2,753ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-01-03 08:28:34 [2,753ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-01-03 08:28:34 [2,753ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-01-03 08:28:34 [2,753ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-01-03 08:28:34 [2,753ms] [Info] [omni.meta-upgrade] [6291] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-01-03 08:28:34 [2,795ms] [Info] [omni.core] [6291] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-01-03 08:28:34 [2,796ms] [Info] [omni.core] [6291] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-01-03 08:28:34 [2,799ms] [Info] [omni.core] [6291] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-01-03 08:28:34 [2,799ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-01-03 08:28:34 [2,799ms] [Info] [omni.core] [6297] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-01-03 08:28:34 [2,799ms] [Info] [omni.core] [6299] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-01-03 08:28:34 [2,799ms] [Info] [carb.launcher.plugin] [6299] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7f48b40025a0 for './omni.delta-applier.app'. 2023-01-03 08:28:34 [2,799ms] [Info] [carb.launcher.plugin] [6299] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 6291. 2023-01-03 08:28:34 [2,801ms] [Info] [carb.launcher.plugin] [6306] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 6359 from the parent 6291. 2023-01-03 08:28:34 [2,801ms] [Info] [carb.launcher.plugin] [6306] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 6359. 2023-01-03 08:28:34 [2,803ms] [Info] [carb.launcher.plugin] [6306] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 6359. 2023-01-03 08:28:34 [2,803ms] [Info] [carb.launcher.plugin] [6299] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 6359 from the parent 6291. 2023-01-03 08:28:34 [2,803ms] [Info] [carb.launcher.plugin] [6299] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 6359 {process = 0x7f48b40025a0} 2023-01-03 08:28:34 [2,803ms] [Info] [carb.launcher.plugin] [6360] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 6359. 2023-01-03 08:28:34 [2,803ms] [Info] [carb.launcher.plugin] [6361] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 6359. 2023-01-03 08:28:35 [3,005ms] [Info] [omni.core] [6298] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-01-03 08:28:35 [3,607ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 17:28:35,664 - system_log - INFO - reconnecting ... 2023-01-03 08:28:35 [3,608ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:28:35,665 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 08:28:36 [4,610ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 17:28:36,666 - system_log - INFO - reconnecting ... 2023-01-03 08:28:36 [4,611ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:28:36,668 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 08:28:36 [4,799ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-01-03 08:28:36 [4,800ms] [Info] [omni.server] [6291] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-01-03 08:28:36 [4,800ms] [Info] [omni.signalhandler] [6291] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-01-03 08:28:36 [4,800ms] [Info] [omni.signalhandler] [6291] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-01-03 08:28:36 [4,800ms] [Info] [omni.signalhandler] [6291] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-01-03 08:28:37 [5,612ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 17:28:37,669 - system_log - INFO - reconnecting ... 2023-01-03 08:28:37 [5,615ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:28:37,671 - system_log - INFO - connected 2023-01-03 08:28:37 [5,616ms] [Error] [omni.core] [6309] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-01-03 08:28:46 [0ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-03 08:28:46 [0ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 08:28:46 [0ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 08:28:46 [0ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 08:28:46 [0ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 08:28:46 [0ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-03 08:28:46 [0ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6533] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6533] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6533] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6533] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6533] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6533] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6533] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6533] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6533] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6533] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6533] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "0.0.0.0" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "0.0.0.0" } }, "S3": { "RESOLVER": { "CACHE": "" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "0.0.0.0", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-03 08:28:46 [2ms] [Info] [carb] [6533] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 08:28:46 [2ms] [Info] [carb] [6533] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 08:28:46 [3ms] [Info] [carb] [6533] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 08:28:46 [3ms] [Info] [carb] [6533] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 08:28:46 [3ms] [Info] [carb.launcher.plugin] [6533] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1fc9a30 for './omni-lft-service/omni-lft-service'. 2023-01-03 08:28:46 [3ms] [Info] [carb.launcher.plugin] [6533] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 6533. 2023-01-03 08:28:46 [3ms] [Info] [carb] [6533] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-03 08:28:46 [3ms] [Info] [carb] [6533] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-03 08:28:46 [4ms] [Info] [carb.launcher.plugin] [6554] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 6555 from the parent 6533. 2023-01-03 08:28:46 [4ms] [Info] [carb.launcher.plugin] [6554] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 6555. 2023-01-03 08:28:46 [4ms] [Info] [carb.launcher.plugin] [6554] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 6555. 2023-01-03 08:28:46 [4ms] [Info] [carb.launcher.plugin] [6533] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 6555 from the parent 6533. 2023-01-03 08:28:46 [5ms] [Info] [carb.launcher.plugin] [6533] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 6555 {process = 0x1fc9a30} 2023-01-03 08:28:46 [5ms] [Info] [omni.core] [6533] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-03 08:28:46 [5ms] [Info] [omni.core] [6533] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-03 08:28:46 [5ms] [Info] [carb.launcher.plugin] [6560] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 6555. 2023-01-03 08:28:46 [5ms] [Info] [carb.launcher.plugin] [6559] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 6555. 2023-01-03 08:28:46 [0ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-03 08:28:46 [0ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 08:28:46 [0ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 08:28:46 [0ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 08:28:46 [0ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-03 08:28:46 [1ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 08:28:46 [2ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-03 08:28:46 [2ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 08:28:46 [2ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 08:28:46 [2ms] [Info] [carb] [6562] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 08:28:46 [2ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 08:28:46 [2ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 08:28:46 [2ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 08:28:46 [2ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 08:28:46 [2ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6562] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6562] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-03 08:28:46 [2ms] [Info] [omni.server] [6562] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-03 08:28:46 [3ms] [Info] [omni.server] [6562] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-03 08:28:46 [3ms] [Info] [omni.server] [6562] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-03 08:28:46 [3ms] [Info] [omni.server] [6562] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-03 08:28:46 [3ms] [Info] [omni.server] [6562] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-03 08:28:46 [3ms] [Info] [omni.server] [6562] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-03 08:28:46 [3ms] [Info] [omni.server] [6562] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-03 08:28:46 [3ms] [Info] [omni.server] [6562] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "0.0.0.0" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "0.0.0.0" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "0.0.0.0", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-03 08:28:46 [3ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 08:28:46 [4ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 08:28:46 [4ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 08:28:46 [5ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 08:28:46 [5ms] [Info] [carb.launcher.plugin] [6562] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x166da40 for './omni-lft-service/omni-lft-service'. 2023-01-03 08:28:46 [5ms] [Info] [carb.launcher.plugin] [6562] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 6562. 2023-01-03 08:28:46 [5ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-03 08:28:46 [5ms] [Info] [carb] [6562] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-03 08:28:46 [5ms] [Info] [carb.launcher.plugin] [6574] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 6575 from the parent 6562. 2023-01-03 08:28:46 [5ms] [Info] [carb.launcher.plugin] [6574] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 6575. 2023-01-03 08:28:46 [6ms] [Info] [carb.launcher.plugin] [6574] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 6575. 2023-01-03 08:28:46 [6ms] [Info] [carb.launcher.plugin] [6562] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 6575 from the parent 6562. 2023-01-03 08:28:46 [6ms] [Info] [carb.launcher.plugin] [6562] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 6575 {process = 0x166da40} 2023-01-03 08:28:46 [6ms] [Info] [omni.core] [6562] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-03 08:28:46 [6ms] [Info] [omni.core] [6562] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-03 08:28:46 [6ms] [Info] [carb.launcher.plugin] [6578] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 6575. 2023-01-03 08:28:46 [7ms] [Info] [carb.launcher.plugin] [6577] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 6575. 2023-01-03 08:28:46 [284ms] [Error] [omni.core] [6578] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-03 08:28:46 [287ms] [Error] [omni.core] [6578] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:28:46,752 - system_log - INFO - reconnecting ... 2023-01-03 08:28:46 [296ms] [Error] [omni.core] [6578] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:28:46,761 - system_log - INFO - connected 2023-01-03 08:28:46 [297ms] [Error] [omni.core] [6578] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('0.0.0.0', 3030): address already in use [6575] Failed to execute script 'run_server' due to unhandled exception! 2023-01-03 08:28:46 [327ms] [Info] [carb.launcher.plugin] [6577] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-03 08:28:46 [327ms] [Info] [carb.launcher.plugin] [6578] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-03 08:28:46 [327ms] [Info] [carb.launcher.plugin] [6577] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 6575, bytes = 0} 2023-01-03 08:28:46 [327ms] [Info] [carb.launcher.plugin] [6578] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 6575, bytes = 0} 2023-01-03 08:28:46 [327ms] [Info] [carb.launcher.plugin] [6578] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 6575. 2023-01-03 08:28:46 [327ms] [Info] [carb.launcher.plugin] [6577] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 6575. 2023-01-03 08:28:48 [2,006ms] [Info] [omni.core] [6567] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-03 08:28:48 [2,006ms] [Info] [omni.core] [6567] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-03 08:28:48 [2,006ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 08:28:48 [2,007ms] [Info] [omni.core] [6562] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-03 08:28:48 [2,008ms] [Info] [carb] [6562] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 08:28:48 [2,008ms] [Warning] [omni.keyvaluedatabase.leveldb] [6562] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-03 08:28:48 [2,008ms] [Info] [omni.core] [6567] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-03 08:28:48 [2,046ms] [Warning] [omni.keyvaluedatabase.leveldb] [6562] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-03 08:28:48 [2,046ms] [Fatal] [omni.core] [6562] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-03 08:33:59 [0ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8558] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 08:33:59 [2ms] [Info] [omni.server] [8558] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-03 08:33:59 [2ms] [Info] [omni.server] [8558] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-03 08:33:59 [2ms] [Info] [omni.server] [8558] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-03 08:33:59 [2ms] [Info] [omni.server] [8558] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-03 08:33:59 [2ms] [Info] [omni.server] [8558] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-03 08:33:59 [2ms] [Info] [omni.server] [8558] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-03 08:33:59 [2ms] [Info] [omni.server] [8558] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-03 08:33:59 [2ms] [Info] [omni.server] [8558] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-03 08:33:59 [2ms] [Info] [omni.server] [8558] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-03 08:33:59 [2ms] [Info] [omni.server] [8558] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-03 08:33:59 [2ms] [Info] [carb] [8558] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 08:33:59 [3ms] [Info] [carb] [8558] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 08:33:59 [3ms] [Info] [carb] [8558] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 08:33:59 [3ms] [Info] [carb] [8558] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 08:33:59 [3ms] [Info] [carb.launcher.plugin] [8558] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1e62550 for './omni-lft-service/omni-lft-service'. 2023-01-03 08:33:59 [3ms] [Info] [carb.launcher.plugin] [8558] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 8558. 2023-01-03 08:33:59 [4ms] [Info] [carb] [8558] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-03 08:33:59 [4ms] [Info] [carb] [8558] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-03 08:33:59 [6ms] [Info] [carb.launcher.plugin] [8582] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 8583 from the parent 8558. 2023-01-03 08:33:59 [6ms] [Info] [carb.launcher.plugin] [8582] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 8583. 2023-01-03 08:33:59 [8ms] [Info] [carb.launcher.plugin] [8582] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 8583. 2023-01-03 08:33:59 [8ms] [Info] [carb.launcher.plugin] [8558] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 8583 from the parent 8558. 2023-01-03 08:33:59 [8ms] [Info] [carb.launcher.plugin] [8558] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 8583 {process = 0x1e62550} 2023-01-03 08:33:59 [9ms] [Info] [carb.launcher.plugin] [8585] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 8583. 2023-01-03 08:33:59 [9ms] [Info] [omni.core] [8558] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-03 08:33:59 [9ms] [Info] [omni.core] [8558] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-03 08:33:59 [0ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-03 08:33:59 [0ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-03 08:33:59 [1ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-03 08:33:59 [5ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-03 08:33:59 [5ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-03 08:33:59 [5ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-03 08:33:59 [5ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-03 08:33:59 [5ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-03 08:33:59 [5ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-03 08:33:59 [5ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-03 08:33:59 [5ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-03 08:33:59 [5ms] [Info] [omni.server] [8587] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-03 08:33:59 [5ms] [Info] [omni.server] [8587] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-03 08:33:59 [5ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-03 08:33:59 [5ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-03 08:33:59 [6ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-03 08:33:59 [6ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-03 08:33:59 [6ms] [Info] [carb.launcher.plugin] [8587] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x10b7540 for './omni-lft-service/omni-lft-service'. 2023-01-03 08:33:59 [6ms] [Info] [carb.launcher.plugin] [8587] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 8587. 2023-01-03 08:33:59 [6ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-03 08:33:59 [6ms] [Info] [carb] [8587] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-03 08:33:59 [7ms] [Info] [carb.launcher.plugin] [8599] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 8601 from the parent 8587. 2023-01-03 08:33:59 [7ms] [Info] [carb.launcher.plugin] [8599] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 8601. 2023-01-03 08:33:59 [8ms] [Info] [carb.launcher.plugin] [8599] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 8601. 2023-01-03 08:33:59 [8ms] [Info] [carb.launcher.plugin] [8587] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 8601 from the parent 8587. 2023-01-03 08:33:59 [8ms] [Info] [carb.launcher.plugin] [8587] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 8601 {process = 0x10b7540} 2023-01-03 08:33:59 [8ms] [Info] [omni.core] [8587] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-03 08:33:59 [8ms] [Info] [omni.core] [8587] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-03 08:33:59 [8ms] [Info] [carb.launcher.plugin] [8602] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 8601. 2023-01-03 08:33:59 [8ms] [Info] [carb.launcher.plugin] [8603] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 8601. 2023-01-03 08:33:59 [270ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-03 08:33:59 [273ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:33:59,421 - system_log - INFO - reconnecting ... 2023-01-03 08:33:59 [281ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:33:59,428 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 08:34:00 [1,282ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 17:34:00,430 - system_log - INFO - reconnecting ... 2023-01-03 08:34:00 [1,285ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:34:00,432 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 08:34:01 [2,008ms] [Info] [omni.core] [8592] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-03 08:34:01 [2,008ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-03 08:34:01 [2,008ms] [Info] [omni.core] [8592] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-03 08:34:01 [2,008ms] [Info] [omni.core] [8587] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-03 08:34:01 [2,008ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-03 08:34:01 [2,009ms] [Info] [omni.core] [8592] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-03 08:34:01 [2,077ms] [Warning] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-03 08:34:01 [2,077ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-03 08:34:01 [2,121ms] [Warning] [carb] [8587] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-03 08:34:01 [2,121ms] [Info] [carb] [8587] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-03 08:34:01 [2,121ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-01-03 08:34:01 [2,147ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-01-03 08:34:01 [2,147ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 17536 records 2023-01-03 08:34:01 [2,154ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1001 / 17536 records 2023-01-03 08:34:01 [2,157ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 2001 / 17536 records 2023-01-03 08:34:01 [2,159ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 3001 / 17536 records 2023-01-03 08:34:01 [2,161ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 4001 / 17536 records 2023-01-03 08:34:01 [2,163ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 5001 / 17536 records 2023-01-03 08:34:01 [2,165ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 6001 / 17536 records 2023-01-03 08:34:01 [2,167ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 7001 / 17536 records 2023-01-03 08:34:01 [2,168ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 8001 / 17536 records 2023-01-03 08:34:01 [2,168ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 9001 / 17536 records 2023-01-03 08:34:01 [2,168ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 10001 / 17536 records 2023-01-03 08:34:01 [2,170ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 11001 / 17536 records 2023-01-03 08:34:01 [2,170ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 12001 / 17536 records 2023-01-03 08:34:01 [2,170ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 13001 / 17536 records 2023-01-03 08:34:01 [2,170ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 14001 / 17536 records 2023-01-03 08:34:01 [2,171ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 15001 / 17536 records 2023-01-03 08:34:01 [2,172ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 16001 / 17536 records 2023-01-03 08:34:01 [2,173ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 17001 / 17536 records 2023-01-03 08:34:01 [2,173ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-01-03 08:34:01 [2,174ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 4242 records 2023-01-03 08:34:01 [2,175ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1001 / 4242 records 2023-01-03 08:34:01 [2,177ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 2001 / 4242 records 2023-01-03 08:34:01 [2,178ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 3001 / 4242 records 2023-01-03 08:34:01 [2,179ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 4001 / 4242 records 2023-01-03 08:34:01 [2,179ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-01-03 08:34:01 [2,186ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-01-03 08:34:01 [2,186ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-01-03 08:34:01 [2,186ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 17536 records 2023-01-03 08:34:01 [2,186ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1001 / 17536 records 2023-01-03 08:34:01 [2,188ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 2001 / 17536 records 2023-01-03 08:34:01 [2,191ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 3001 / 17536 records 2023-01-03 08:34:01 [2,194ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 4001 / 17536 records 2023-01-03 08:34:01 [2,198ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 5001 / 17536 records 2023-01-03 08:34:01 [2,201ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 6001 / 17536 records 2023-01-03 08:34:01 [2,206ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 7001 / 17536 records 2023-01-03 08:34:01 [2,213ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 8001 / 17536 records 2023-01-03 08:34:01 [2,219ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 9001 / 17536 records 2023-01-03 08:34:01 [2,225ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 10001 / 17536 records 2023-01-03 08:34:01 [2,232ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 11001 / 17536 records 2023-01-03 08:34:01 [2,239ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 12001 / 17536 records 2023-01-03 08:34:01 [2,246ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 13001 / 17536 records 2023-01-03 08:34:01 [2,253ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 14001 / 17536 records 2023-01-03 08:34:01 [2,260ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 15001 / 17536 records 2023-01-03 08:34:01 [2,265ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 16001 / 17536 records 2023-01-03 08:34:01 [2,271ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 17001 / 17536 records 2023-01-03 08:34:01 [2,274ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-01-03 08:34:01 [2,274ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-01-03 08:34:01 [2,274ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1 / 7697 records 2023-01-03 08:34:01 [2,275ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1001 / 7697 records 2023-01-03 08:34:01 [2,276ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 2001 / 7697 records 2023-01-03 08:34:01 [2,277ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 3001 / 7697 records 2023-01-03 08:34:01 [2,278ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 4001 / 7697 records 2023-01-03 08:34:01 [2,278ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 5001 / 7697 records 2023-01-03 08:34:01 [2,279ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 6001 / 7697 records 2023-01-03 08:34:01 [2,280ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 7001 / 7697 records 2023-01-03 08:34:01 [2,281ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-01-03 08:34:01 [2,281ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-01-03 08:34:01 [2,287ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 17:34:01,435 - system_log - INFO - reconnecting ... 2023-01-03 08:34:01 [2,287ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:34:01,435 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 08:34:01 [2,290ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-01-03 08:34:01 [2,290ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-01-03 08:34:01 [2,290ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-03 08:34:01 [2,290ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-03 08:34:01 [2,290ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-03 08:34:01 [2,290ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-03 08:34:01 [2,290ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-03 08:34:01 [2,290ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-03 08:34:01 [2,291ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-03 08:34:01 [2,291ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-03 08:34:01 [2,291ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-03 08:34:01 [2,291ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-03 08:34:01 [2,292ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-03 08:34:01 [2,293ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-03 08:34:01 [2,293ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-03 08:34:01 [2,293ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-03 08:34:01 [2,293ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-03 08:34:01 [2,293ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-03 08:34:01 [2,294ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-03 08:34:01 [2,301ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-03 08:34:01 [2,307ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-03 08:34:01 [2,313ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-03 08:34:01 [2,319ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-03 08:34:01 [2,326ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-03 08:34:01 [2,332ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-03 08:34:01 [2,340ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-03 08:34:01 [2,349ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-03 08:34:01 [2,357ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-03 08:34:01 [2,365ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-03 08:34:01 [2,372ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-03 08:34:01 [2,381ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-03 08:34:01 [2,389ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-03 08:34:01 [2,397ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-03 08:34:01 [2,406ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-03 08:34:01 [2,414ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-03 08:34:01 [2,421ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-03 08:34:01 [2,423ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-01-03 08:34:01 [2,450ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-01-03 08:34:01 [2,450ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-01-03 08:34:01 [2,456ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-01-03 08:34:01 [2,457ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-01-03 08:34:01 [2,457ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-01-03 08:34:01 [2,457ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-01-03 08:34:01 [2,457ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-01-03 08:34:01 [2,457ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-01-03 08:34:01 [2,457ms] [Info] [omni.meta-upgrade] [8587] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-01-03 08:34:01 [2,472ms] [Info] [omni.core] [8587] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-01-03 08:34:01 [2,472ms] [Info] [omni.core] [8587] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-01-03 08:34:01 [2,474ms] [Info] [omni.core] [8587] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-01-03 08:34:01 [2,474ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-01-03 08:34:01 [2,474ms] [Info] [omni.core] [8591] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-01-03 08:34:01 [2,475ms] [Info] [omni.core] [8592] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-01-03 08:34:01 [2,475ms] [Info] [carb.launcher.plugin] [8592] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7efea0006700 for './omni.delta-applier.app'. 2023-01-03 08:34:01 [2,475ms] [Info] [carb.launcher.plugin] [8592] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 8587. 2023-01-03 08:34:01 [2,476ms] [Info] [carb.launcher.plugin] [8599] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 8644 from the parent 8587. 2023-01-03 08:34:01 [2,476ms] [Info] [carb.launcher.plugin] [8599] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 8644. 2023-01-03 08:34:01 [2,477ms] [Info] [carb.launcher.plugin] [8599] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 8644. 2023-01-03 08:34:01 [2,477ms] [Info] [carb.launcher.plugin] [8592] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 8644 from the parent 8587. 2023-01-03 08:34:01 [2,477ms] [Info] [carb.launcher.plugin] [8592] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 8644 {process = 0x7efea0006700} 2023-01-03 08:34:01 [2,477ms] [Info] [carb.launcher.plugin] [8645] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 8644. 2023-01-03 08:34:01 [2,477ms] [Info] [carb.launcher.plugin] [8646] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 8644. 2023-01-03 08:34:01 [2,678ms] [Info] [omni.core] [8593] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-01-03 08:34:02 [3,288ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 17:34:02,436 - system_log - INFO - reconnecting ... 2023-01-03 08:34:02 [3,290ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:34:02,438 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 08:34:03 [4,293ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 17:34:03,440 - system_log - INFO - reconnecting ... 2023-01-03 08:34:03 [4,295ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:34:03,443 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 08:34:03 [4,475ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-01-03 08:34:03 [4,477ms] [Info] [omni.server] [8587] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-01-03 08:34:03 [4,477ms] [Info] [omni.signalhandler] [8587] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-01-03 08:34:03 [4,478ms] [Info] [omni.signalhandler] [8587] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-01-03 08:34:03 [4,478ms] [Info] [omni.signalhandler] [8587] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-01-03 08:34:04 [5,297ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-03 17:34:04,445 - system_log - INFO - reconnecting ... 2023-01-03 08:34:04 [5,302ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-03 17:34:04,450 - system_log - INFO - connected 2023-01-03 08:34:04 [5,303ms] [Error] [omni.core] [8603] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-01-04 04:29:58 [0ms] [Info] [carb] [3146] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-04 04:29:58 [1ms] [Info] [carb] [3146] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-04 04:29:58 [4ms] [Info] [carb] [3146] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-04 04:29:58 [5ms] [Info] [carb] [3146] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-04 04:29:58 [0ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-04 04:29:58 [0ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-04 04:29:58 [0ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-04 04:29:58 [0ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-04 04:29:58 [3ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-04 04:29:58 [5ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-04 04:29:58 [5ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-04 04:29:58 [6ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-04 04:29:58 [6ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-04 04:29:58 [7ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-04 04:29:58 [9ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-04 04:29:58 [9ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-04 04:29:58 [11ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-04 04:29:58 [12ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-04 04:29:58 [12ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-04 04:29:58 [14ms] [Info] [carb] [3162] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-04 04:29:58 [14ms] [Info] [carb] [3162] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-04 04:29:58 [14ms] [Info] [carb] [3162] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-04 04:29:58 [14ms] [Info] [carb] [3162] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-04 04:29:58 [14ms] [Info] [carb] [3162] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-04 04:29:58 [14ms] [Info] [carb] [3162] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-04 04:29:58 [15ms] [Info] [omni.server] [3162] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-04 04:29:58 [15ms] [Info] [omni.server] [3162] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-04 04:29:58 [15ms] [Info] [omni.server] [3162] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-04 04:29:58 [16ms] [Info] [omni.server] [3162] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-04 04:29:58 [16ms] [Info] [omni.server] [3162] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-04 04:29:58 [16ms] [Info] [omni.server] [3162] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-04 04:29:58 [16ms] [Info] [omni.server] [3162] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-04 04:29:58 [16ms] [Info] [omni.server] [3162] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-04 04:29:58 [16ms] [Info] [omni.server] [3162] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-04 04:29:58 [16ms] [Info] [omni.server] [3162] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-04 04:29:58 [16ms] [Info] [carb] [3162] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-04 04:29:58 [17ms] [Info] [carb] [3162] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-04 04:29:58 [18ms] [Info] [carb] [3162] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-04 04:29:58 [19ms] [Info] [carb] [3162] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-04 04:29:58 [19ms] [Info] [carb.launcher.plugin] [3162] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1df0e60 for './omni-lft-service/omni-lft-service'. 2023-01-04 04:29:58 [19ms] [Info] [carb.launcher.plugin] [3162] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3162. 2023-01-04 04:29:58 [19ms] [Info] [carb] [3162] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-04 04:29:58 [19ms] [Info] [carb] [3162] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-04 04:29:58 [19ms] [Info] [carb.launcher.plugin] [3180] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3181 from the parent 3162. 2023-01-04 04:29:58 [19ms] [Info] [carb.launcher.plugin] [3180] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3181. 2023-01-04 04:29:58 [20ms] [Info] [carb.launcher.plugin] [3180] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3181. 2023-01-04 04:29:58 [20ms] [Info] [carb.launcher.plugin] [3162] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3181 from the parent 3162. 2023-01-04 04:29:58 [20ms] [Info] [carb.launcher.plugin] [3182] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3181. 2023-01-04 04:29:58 [20ms] [Info] [carb.launcher.plugin] [3162] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3181 {process = 0x1df0e60} 2023-01-04 04:29:58 [20ms] [Info] [omni.core] [3162] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-04 04:29:58 [20ms] [Info] [omni.core] [3162] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-04 04:29:58 [20ms] [Info] [carb.launcher.plugin] [3183] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3181. 2023-01-04 04:29:59 [427ms] [Error] [omni.core] [3183] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-04 04:29:59 [435ms] [Error] [omni.core] [3183] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-04 13:29:59,087 - system_log - INFO - reconnecting ... 2023-01-04 04:29:59 [445ms] [Error] [omni.core] [3183] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-04 13:29:59,097 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 04:30:00 [0ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-04 04:30:00 [0ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-04 04:30:00 [0ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-04 04:30:00 [0ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-04 04:30:00 [0ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-04 04:30:00 [0ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-04 04:30:00 [0ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-04 04:30:00 [0ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-04 04:30:00 [0ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-04 04:30:00 [0ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-04 04:30:00 [1ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-04 04:30:00 [1ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-04 04:30:00 [1ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-04 04:30:00 [1ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-04 04:30:00 [1ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-04 04:30:00 [1ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-04 04:30:00 [1ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-04 04:30:00 [1ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-04 04:30:00 [1ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-04 04:30:00 [1ms] [Info] [omni.server] [3474] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-04 04:30:00 [2ms] [Info] [omni.server] [3474] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-04 04:30:00 [2ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-04 04:30:00 [2ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-04 04:30:00 [2ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-04 04:30:00 [2ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-04 04:30:00 [2ms] [Info] [carb.launcher.plugin] [3474] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1828a90 for './omni-lft-service/omni-lft-service'. 2023-01-04 04:30:00 [2ms] [Info] [carb.launcher.plugin] [3474] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3474. 2023-01-04 04:30:00 [2ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-04 04:30:00 [2ms] [Info] [carb] [3474] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-04 04:30:00 [3ms] [Info] [carb.launcher.plugin] [3489] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3490 from the parent 3474. 2023-01-04 04:30:00 [3ms] [Info] [carb.launcher.plugin] [3489] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3490. 2023-01-04 04:30:00 [3ms] [Info] [carb.launcher.plugin] [3489] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3490. 2023-01-04 04:30:00 [3ms] [Info] [carb.launcher.plugin] [3474] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3490 from the parent 3474. 2023-01-04 04:30:00 [3ms] [Info] [carb.launcher.plugin] [3474] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3490 {process = 0x1828a90} 2023-01-04 04:30:00 [3ms] [Info] [carb.launcher.plugin] [3491] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3490. 2023-01-04 04:30:00 [3ms] [Info] [omni.core] [3474] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-04 04:30:00 [3ms] [Info] [carb.launcher.plugin] [3492] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3490. 2023-01-04 04:30:00 [3ms] [Info] [omni.core] [3474] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-04 04:30:00 [237ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-04 04:30:00 [240ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-04 13:30:00,454 - system_log - INFO - reconnecting ... 2023-01-04 04:30:00 [248ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-04 13:30:00,461 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 04:30:01 [1,249ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 13:30:01,463 - system_log - INFO - reconnecting ... 2023-01-04 04:30:01 [1,250ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-04 13:30:01,463 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 04:30:02 [2,003ms] [Info] [omni.core] [3481] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-04 04:30:02 [2,003ms] [Info] [omni.core] [3481] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-04 04:30:02 [2,004ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-04 04:30:02 [2,004ms] [Info] [omni.core] [3481] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-04 04:30:02 [2,008ms] [Info] [omni.core] [3474] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-04 04:30:02 [2,008ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-04 04:30:02 [2,092ms] [Warning] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-04 04:30:02 [2,092ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-04 04:30:02 [2,122ms] [Warning] [carb] [3474] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-04 04:30:02 [2,122ms] [Info] [carb] [3474] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-04 04:30:02 [2,122ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-01-04 04:30:02 [2,131ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-01-04 04:30:02 [2,131ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 17536 records 2023-01-04 04:30:02 [2,136ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1001 / 17536 records 2023-01-04 04:30:02 [2,139ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 2001 / 17536 records 2023-01-04 04:30:02 [2,142ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 3001 / 17536 records 2023-01-04 04:30:02 [2,144ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 4001 / 17536 records 2023-01-04 04:30:02 [2,147ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 5001 / 17536 records 2023-01-04 04:30:02 [2,149ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 6001 / 17536 records 2023-01-04 04:30:02 [2,151ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 7001 / 17536 records 2023-01-04 04:30:02 [2,151ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 8001 / 17536 records 2023-01-04 04:30:02 [2,152ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 9001 / 17536 records 2023-01-04 04:30:02 [2,152ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 10001 / 17536 records 2023-01-04 04:30:02 [2,153ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 11001 / 17536 records 2023-01-04 04:30:02 [2,153ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 12001 / 17536 records 2023-01-04 04:30:02 [2,153ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 13001 / 17536 records 2023-01-04 04:30:02 [2,154ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 14001 / 17536 records 2023-01-04 04:30:02 [2,154ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 15001 / 17536 records 2023-01-04 04:30:02 [2,155ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 16001 / 17536 records 2023-01-04 04:30:02 [2,156ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 17001 / 17536 records 2023-01-04 04:30:02 [2,156ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-01-04 04:30:02 [2,158ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 4242 records 2023-01-04 04:30:02 [2,160ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1001 / 4242 records 2023-01-04 04:30:02 [2,162ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 2001 / 4242 records 2023-01-04 04:30:02 [2,164ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 3001 / 4242 records 2023-01-04 04:30:02 [2,166ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 4001 / 4242 records 2023-01-04 04:30:02 [2,166ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-01-04 04:30:02 [2,172ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-01-04 04:30:02 [2,172ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-01-04 04:30:02 [2,172ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 17536 records 2023-01-04 04:30:02 [2,172ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1001 / 17536 records 2023-01-04 04:30:02 [2,175ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 2001 / 17536 records 2023-01-04 04:30:02 [2,178ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 3001 / 17536 records 2023-01-04 04:30:02 [2,182ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 4001 / 17536 records 2023-01-04 04:30:02 [2,186ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 5001 / 17536 records 2023-01-04 04:30:02 [2,190ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 6001 / 17536 records 2023-01-04 04:30:02 [2,196ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 7001 / 17536 records 2023-01-04 04:30:02 [2,203ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 8001 / 17536 records 2023-01-04 04:30:02 [2,210ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 9001 / 17536 records 2023-01-04 04:30:02 [2,217ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 10001 / 17536 records 2023-01-04 04:30:02 [2,225ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 11001 / 17536 records 2023-01-04 04:30:02 [2,231ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 12001 / 17536 records 2023-01-04 04:30:02 [2,238ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 13001 / 17536 records 2023-01-04 04:30:02 [2,245ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 14001 / 17536 records 2023-01-04 04:30:02 [2,251ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 13:30:02,464 - system_log - INFO - reconnecting ... 2023-01-04 04:30:02 [2,251ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 15001 / 17536 records 2023-01-04 04:30:02 [2,252ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-04 13:30:02,465 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 04:30:02 [2,257ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 16001 / 17536 records 2023-01-04 04:30:02 [2,263ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 17001 / 17536 records 2023-01-04 04:30:02 [2,266ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-01-04 04:30:02 [2,266ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-01-04 04:30:02 [2,266ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1 / 7697 records 2023-01-04 04:30:02 [2,267ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1001 / 7697 records 2023-01-04 04:30:02 [2,268ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 2001 / 7697 records 2023-01-04 04:30:02 [2,269ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 3001 / 7697 records 2023-01-04 04:30:02 [2,270ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 4001 / 7697 records 2023-01-04 04:30:02 [2,271ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 5001 / 7697 records 2023-01-04 04:30:02 [2,272ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 6001 / 7697 records 2023-01-04 04:30:02 [2,273ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 7001 / 7697 records 2023-01-04 04:30:02 [2,273ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-01-04 04:30:02 [2,274ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-01-04 04:30:02 [2,282ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-01-04 04:30:02 [2,282ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-01-04 04:30:02 [2,282ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-04 04:30:02 [2,282ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-04 04:30:02 [2,282ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-04 04:30:02 [2,282ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-04 04:30:02 [2,282ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-04 04:30:02 [2,282ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-04 04:30:02 [2,282ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-04 04:30:02 [2,283ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-04 04:30:02 [2,283ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-04 04:30:02 [2,283ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-04 04:30:02 [2,284ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-04 04:30:02 [2,284ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-04 04:30:02 [2,284ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-04 04:30:02 [2,285ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-04 04:30:02 [2,285ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-04 04:30:02 [2,285ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-04 04:30:02 [2,286ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-04 04:30:02 [2,293ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-04 04:30:02 [2,300ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-04 04:30:02 [2,306ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-04 04:30:02 [2,313ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-04 04:30:02 [2,319ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-04 04:30:02 [2,326ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-04 04:30:02 [2,334ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-04 04:30:02 [2,345ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-04 04:30:02 [2,354ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-04 04:30:02 [2,364ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-04 04:30:02 [2,372ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-04 04:30:02 [2,383ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-04 04:30:02 [2,392ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-04 04:30:02 [2,401ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-04 04:30:02 [2,410ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-04 04:30:02 [2,420ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-04 04:30:02 [2,427ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-04 04:30:02 [2,430ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-01-04 04:30:02 [2,461ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-01-04 04:30:02 [2,461ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-01-04 04:30:02 [2,468ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-01-04 04:30:02 [2,469ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-01-04 04:30:02 [2,469ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-01-04 04:30:02 [2,469ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-01-04 04:30:02 [2,469ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-01-04 04:30:02 [2,469ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-01-04 04:30:02 [2,469ms] [Info] [omni.meta-upgrade] [3474] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-01-04 04:30:02 [2,487ms] [Info] [omni.core] [3474] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-01-04 04:30:02 [2,490ms] [Info] [omni.core] [3474] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-01-04 04:30:02 [2,492ms] [Info] [omni.core] [3474] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-01-04 04:30:02 [2,492ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-01-04 04:30:02 [2,492ms] [Info] [omni.core] [3482] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-01-04 04:30:02 [2,493ms] [Info] [omni.core] [3481] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-01-04 04:30:02 [2,493ms] [Info] [carb.launcher.plugin] [3481] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7f4c880053b0 for './omni.delta-applier.app'. 2023-01-04 04:30:02 [2,493ms] [Info] [carb.launcher.plugin] [3481] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 3474. 2023-01-04 04:30:02 [2,493ms] [Info] [carb.launcher.plugin] [3489] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4044 from the parent 3474. 2023-01-04 04:30:02 [2,493ms] [Info] [carb.launcher.plugin] [3489] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4044. 2023-01-04 04:30:02 [2,494ms] [Info] [carb.launcher.plugin] [3489] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4044. 2023-01-04 04:30:02 [2,494ms] [Info] [carb.launcher.plugin] [3481] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4044 from the parent 3474. 2023-01-04 04:30:02 [2,495ms] [Info] [carb.launcher.plugin] [3481] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4044 {process = 0x7f4c880053b0} 2023-01-04 04:30:02 [2,495ms] [Info] [carb.launcher.plugin] [4045] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4044. 2023-01-04 04:30:02 [2,495ms] [Info] [carb.launcher.plugin] [4046] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4044. 2023-01-04 04:30:02 [2,696ms] [Info] [omni.core] [3483] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-01-04 04:30:03 [3,253ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 13:30:03,466 - system_log - INFO - reconnecting ... 2023-01-04 04:30:03 [3,253ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-04 13:30:03,467 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 04:30:04 [4,255ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 13:30:04,468 - system_log - INFO - reconnecting ... 2023-01-04 04:30:04 [4,255ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-04 13:30:04,469 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 04:30:04 [4,492ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-01-04 04:30:04 [4,494ms] [Info] [omni.server] [3474] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-01-04 04:30:04 [4,494ms] [Info] [omni.signalhandler] [3474] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-01-04 04:30:04 [4,494ms] [Info] [omni.signalhandler] [3474] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-01-04 04:30:04 [4,494ms] [Info] [omni.signalhandler] [3474] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-01-04 04:30:05 [5,256ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-04 13:30:05,469 - system_log - INFO - reconnecting ... 2023-01-04 04:30:05 [5,266ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-04 13:30:05,479 - system_log - INFO - connected 2023-01-04 04:30:05 [5,266ms] [Error] [omni.core] [3492] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-01-05 04:47:40 [0ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-05 04:47:40 [0ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-05 04:47:40 [4ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-05 04:47:40 [4ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-05 04:47:40 [9ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-05 04:47:40 [10ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-05 04:47:40 [11ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-05 04:47:40 [12ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-05 04:47:40 [12ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-05 04:47:40 [13ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-05 04:47:40 [15ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-05 04:47:40 [15ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-05 04:47:40 [17ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-05 04:47:40 [17ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-05 04:47:40 [18ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-05 04:47:40 [20ms] [Info] [carb] [3745] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-05 04:47:40 [20ms] [Info] [carb] [3745] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-05 04:47:40 [20ms] [Info] [carb] [3745] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-05 04:47:40 [20ms] [Info] [carb] [3745] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-05 04:47:40 [20ms] [Info] [carb] [3745] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-05 04:47:40 [20ms] [Info] [carb] [3745] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-05 04:47:40 [20ms] [Info] [omni.server] [3745] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-05 04:47:40 [20ms] [Info] [omni.server] [3745] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-05 04:47:40 [20ms] [Info] [omni.server] [3745] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-05 04:47:40 [21ms] [Info] [omni.server] [3745] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-05 04:47:40 [21ms] [Info] [omni.server] [3745] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-05 04:47:40 [21ms] [Info] [omni.server] [3745] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-05 04:47:40 [21ms] [Info] [omni.server] [3745] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-05 04:47:40 [21ms] [Info] [omni.server] [3745] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-05 04:47:40 [21ms] [Info] [omni.server] [3745] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-05 04:47:40 [21ms] [Info] [omni.server] [3745] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-05 04:47:40 [21ms] [Info] [carb] [3745] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-05 04:47:40 [22ms] [Info] [carb] [3745] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-05 04:47:40 [23ms] [Info] [carb] [3745] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-05 04:47:40 [24ms] [Info] [carb] [3745] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-05 04:47:40 [24ms] [Info] [carb.launcher.plugin] [3745] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x20fe4a0 for './omni-lft-service/omni-lft-service'. 2023-01-05 04:47:40 [24ms] [Info] [carb.launcher.plugin] [3745] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3745. 2023-01-05 04:47:40 [24ms] [Info] [carb] [3745] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-05 04:47:40 [24ms] [Info] [carb] [3745] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-05 04:47:40 [25ms] [Info] [carb.launcher.plugin] [3760] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3761 from the parent 3745. 2023-01-05 04:47:40 [25ms] [Info] [carb.launcher.plugin] [3760] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3761. 2023-01-05 04:47:40 [26ms] [Info] [carb.launcher.plugin] [3760] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3761. 2023-01-05 04:47:40 [26ms] [Info] [carb.launcher.plugin] [3745] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3761 from the parent 3745. 2023-01-05 04:47:40 [26ms] [Info] [carb.launcher.plugin] [3745] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3761 {process = 0x20fe4a0} 2023-01-05 04:47:40 [26ms] [Info] [carb.launcher.plugin] [3763] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3761. 2023-01-05 04:47:40 [26ms] [Info] [carb.launcher.plugin] [3762] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3761. 2023-01-05 04:47:40 [26ms] [Info] [omni.core] [3745] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-05 04:47:40 [26ms] [Info] [omni.core] [3745] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-05 04:47:40 [420ms] [Error] [omni.core] [3763] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-05 04:47:40 [426ms] [Error] [omni.core] [3763] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-05 13:47:40,491 - system_log - INFO - reconnecting ... 2023-01-05 04:47:40 [435ms] [Error] [omni.core] [3763] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-05 13:47:40,501 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 04:47:41 [0ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-05 04:47:41 [0ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-05 04:47:41 [0ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-05 04:47:41 [0ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-05 04:47:41 [0ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-05 04:47:41 [0ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-05 04:47:41 [0ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-05 04:47:41 [0ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-05 04:47:41 [0ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-05 04:47:41 [0ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-05 04:47:41 [1ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-05 04:47:41 [1ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-05 04:47:41 [1ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-05 04:47:41 [1ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-05 04:47:41 [1ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-05 04:47:41 [1ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-05 04:47:41 [1ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-05 04:47:41 [1ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-05 04:47:41 [1ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-05 04:47:41 [1ms] [Info] [omni.server] [4070] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-05 04:47:41 [2ms] [Info] [omni.server] [4070] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-05 04:47:41 [2ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-05 04:47:41 [2ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-05 04:47:41 [2ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-05 04:47:41 [2ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-05 04:47:41 [2ms] [Info] [carb.launcher.plugin] [4070] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xbc20c0 for './omni-lft-service/omni-lft-service'. 2023-01-05 04:47:41 [2ms] [Info] [carb.launcher.plugin] [4070] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 4070. 2023-01-05 04:47:41 [2ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-05 04:47:41 [2ms] [Info] [carb] [4070] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-05 04:47:41 [3ms] [Info] [carb.launcher.plugin] [4085] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4086 from the parent 4070. 2023-01-05 04:47:41 [3ms] [Info] [carb.launcher.plugin] [4085] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4086. 2023-01-05 04:47:41 [4ms] [Info] [carb.launcher.plugin] [4085] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4086. 2023-01-05 04:47:41 [4ms] [Info] [carb.launcher.plugin] [4070] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4086 from the parent 4070. 2023-01-05 04:47:41 [4ms] [Info] [carb.launcher.plugin] [4070] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4086 {process = 0xbc20c0} 2023-01-05 04:47:41 [4ms] [Info] [omni.core] [4070] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-05 04:47:41 [4ms] [Info] [carb.launcher.plugin] [4087] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4086. 2023-01-05 04:47:41 [4ms] [Info] [omni.core] [4070] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-05 04:47:41 [4ms] [Info] [carb.launcher.plugin] [4088] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4086. 2023-01-05 04:47:41 [256ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-05 04:47:41 [261ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-05 13:47:41,912 - system_log - INFO - reconnecting ... 2023-01-05 04:47:41 [270ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-05 13:47:41,922 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 04:47:42 [1,272ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 13:47:42,923 - system_log - INFO - reconnecting ... 2023-01-05 04:47:42 [1,272ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-05 13:47:42,924 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 04:47:43 [2,004ms] [Info] [omni.core] [4077] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-05 04:47:43 [2,004ms] [Info] [omni.core] [4077] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-05 04:47:43 [2,005ms] [Info] [omni.core] [4077] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-05 04:47:43 [2,006ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-05 04:47:43 [2,009ms] [Info] [omni.core] [4070] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-05 04:47:43 [2,009ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-05 04:47:43 [2,087ms] [Warning] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-05 04:47:43 [2,087ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-05 04:47:43 [2,116ms] [Warning] [carb] [4070] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-05 04:47:43 [2,116ms] [Info] [carb] [4070] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-05 04:47:43 [2,116ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-01-05 04:47:43 [2,126ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-01-05 04:47:43 [2,126ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 17536 records 2023-01-05 04:47:43 [2,131ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1001 / 17536 records 2023-01-05 04:47:43 [2,134ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 2001 / 17536 records 2023-01-05 04:47:43 [2,136ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 3001 / 17536 records 2023-01-05 04:47:43 [2,138ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 4001 / 17536 records 2023-01-05 04:47:43 [2,141ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 5001 / 17536 records 2023-01-05 04:47:43 [2,143ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 6001 / 17536 records 2023-01-05 04:47:43 [2,145ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 7001 / 17536 records 2023-01-05 04:47:43 [2,145ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 8001 / 17536 records 2023-01-05 04:47:43 [2,146ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 9001 / 17536 records 2023-01-05 04:47:43 [2,146ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 10001 / 17536 records 2023-01-05 04:47:43 [2,147ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 11001 / 17536 records 2023-01-05 04:47:43 [2,147ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 12001 / 17536 records 2023-01-05 04:47:43 [2,147ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 13001 / 17536 records 2023-01-05 04:47:43 [2,147ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 14001 / 17536 records 2023-01-05 04:47:43 [2,148ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 15001 / 17536 records 2023-01-05 04:47:43 [2,149ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 16001 / 17536 records 2023-01-05 04:47:43 [2,150ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 17001 / 17536 records 2023-01-05 04:47:43 [2,150ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-01-05 04:47:43 [2,152ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 4242 records 2023-01-05 04:47:43 [2,154ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1001 / 4242 records 2023-01-05 04:47:43 [2,155ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 2001 / 4242 records 2023-01-05 04:47:43 [2,157ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 3001 / 4242 records 2023-01-05 04:47:43 [2,159ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 4001 / 4242 records 2023-01-05 04:47:43 [2,159ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-01-05 04:47:43 [2,166ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-01-05 04:47:43 [2,166ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-01-05 04:47:43 [2,166ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 17536 records 2023-01-05 04:47:43 [2,166ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1001 / 17536 records 2023-01-05 04:47:43 [2,168ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 2001 / 17536 records 2023-01-05 04:47:43 [2,172ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 3001 / 17536 records 2023-01-05 04:47:43 [2,175ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 4001 / 17536 records 2023-01-05 04:47:43 [2,179ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 5001 / 17536 records 2023-01-05 04:47:43 [2,183ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 6001 / 17536 records 2023-01-05 04:47:43 [2,189ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 7001 / 17536 records 2023-01-05 04:47:43 [2,196ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 8001 / 17536 records 2023-01-05 04:47:43 [2,203ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 9001 / 17536 records 2023-01-05 04:47:43 [2,209ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 10001 / 17536 records 2023-01-05 04:47:43 [2,216ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 11001 / 17536 records 2023-01-05 04:47:43 [2,223ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 12001 / 17536 records 2023-01-05 04:47:43 [2,230ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 13001 / 17536 records 2023-01-05 04:47:43 [2,236ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 14001 / 17536 records 2023-01-05 04:47:43 [2,243ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 15001 / 17536 records 2023-01-05 04:47:43 [2,248ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 16001 / 17536 records 2023-01-05 04:47:43 [2,254ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 17001 / 17536 records 2023-01-05 04:47:43 [2,257ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-01-05 04:47:43 [2,257ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-01-05 04:47:43 [2,257ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1 / 7697 records 2023-01-05 04:47:43 [2,258ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1001 / 7697 records 2023-01-05 04:47:43 [2,259ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 2001 / 7697 records 2023-01-05 04:47:43 [2,260ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 3001 / 7697 records 2023-01-05 04:47:43 [2,261ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 4001 / 7697 records 2023-01-05 04:47:43 [2,262ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 5001 / 7697 records 2023-01-05 04:47:43 [2,263ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 6001 / 7697 records 2023-01-05 04:47:43 [2,263ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 7001 / 7697 records 2023-01-05 04:47:43 [2,264ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-01-05 04:47:43 [2,264ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-01-05 04:47:43 [2,272ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-01-05 04:47:43 [2,272ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-01-05 04:47:43 [2,272ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-05 04:47:43 [2,272ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-05 04:47:43 [2,272ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-05 04:47:43 [2,272ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-05 04:47:43 [2,272ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-05 04:47:43 [2,272ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-05 04:47:43 [2,273ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-05 04:47:43 [2,273ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-05 04:47:43 [2,273ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-05 04:47:43 [2,273ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-05 04:47:43 [2,274ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 13:47:43,925 - system_log - INFO - reconnecting ... 2023-01-05 04:47:43 [2,274ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-05 13:47:43,926 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 04:47:43 [2,274ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-05 04:47:43 [2,274ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-05 04:47:43 [2,275ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-05 04:47:43 [2,275ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-05 04:47:43 [2,275ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-05 04:47:43 [2,275ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-05 04:47:43 [2,276ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-05 04:47:43 [2,283ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-05 04:47:43 [2,289ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-05 04:47:43 [2,295ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-05 04:47:43 [2,302ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-05 04:47:43 [2,308ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-05 04:47:43 [2,314ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-05 04:47:43 [2,322ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-05 04:47:43 [2,330ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-05 04:47:43 [2,338ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-05 04:47:43 [2,347ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-05 04:47:44 [2,354ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-05 04:47:44 [2,363ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-05 04:47:44 [2,371ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-05 04:47:44 [2,379ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-05 04:47:44 [2,388ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-05 04:47:44 [2,398ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-05 04:47:44 [2,405ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-05 04:47:44 [2,407ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-01-05 04:47:44 [2,435ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-01-05 04:47:44 [2,435ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-01-05 04:47:44 [2,441ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-01-05 04:47:44 [2,441ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-01-05 04:47:44 [2,441ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-01-05 04:47:44 [2,441ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-01-05 04:47:44 [2,442ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-01-05 04:47:44 [2,442ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-01-05 04:47:44 [2,442ms] [Info] [omni.meta-upgrade] [4070] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-01-05 04:47:44 [2,456ms] [Info] [omni.core] [4070] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-01-05 04:47:44 [2,458ms] [Info] [omni.core] [4070] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-01-05 04:47:44 [2,460ms] [Info] [omni.core] [4070] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-01-05 04:47:44 [2,461ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-01-05 04:47:44 [2,461ms] [Info] [omni.core] [4078] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-01-05 04:47:44 [2,461ms] [Info] [omni.core] [4080] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-01-05 04:47:44 [2,461ms] [Info] [carb.launcher.plugin] [4080] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7f02a4002500 for './omni.delta-applier.app'. 2023-01-05 04:47:44 [2,461ms] [Info] [carb.launcher.plugin] [4080] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 4070. 2023-01-05 04:47:44 [2,462ms] [Info] [carb.launcher.plugin] [4085] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4621 from the parent 4070. 2023-01-05 04:47:44 [2,462ms] [Info] [carb.launcher.plugin] [4085] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4621. 2023-01-05 04:47:44 [2,463ms] [Info] [carb.launcher.plugin] [4085] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4621. 2023-01-05 04:47:44 [2,463ms] [Info] [carb.launcher.plugin] [4080] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4621 from the parent 4070. 2023-01-05 04:47:44 [2,463ms] [Info] [carb.launcher.plugin] [4080] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4621 {process = 0x7f02a4002500} 2023-01-05 04:47:44 [2,463ms] [Info] [carb.launcher.plugin] [4622] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4621. 2023-01-05 04:47:44 [2,463ms] [Info] [carb.launcher.plugin] [4623] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4621. 2023-01-05 04:47:44 [2,664ms] [Info] [omni.core] [4077] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-01-05 04:47:44 [3,276ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 13:47:44,927 - system_log - INFO - reconnecting ... 2023-01-05 04:47:44 [3,276ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-05 13:47:44,928 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 04:47:45 [4,278ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 13:47:45,929 - system_log - INFO - reconnecting ... 2023-01-05 04:47:45 [4,279ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-05 13:47:45,931 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 04:47:46 [4,461ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-01-05 04:47:46 [4,466ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-01-05 04:47:46 [4,466ms] [Info] [omni.signalhandler] [4070] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-01-05 04:47:46 [4,466ms] [Info] [omni.signalhandler] [4070] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-01-05 04:47:46 [4,466ms] [Info] [omni.signalhandler] [4070] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-01-05 04:47:47 [5,702ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-05 13:47:47,354 - system_log - INFO - reconnecting ... 2023-01-05 04:47:47 [5,706ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-05 13:47:47,357 - system_log - INFO - connected 2023-01-05 04:47:47 [5,706ms] [Error] [omni.core] [4088] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-01-05 05:01:17 [815,795ms] [Warning] [carb] [4077] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::ICore v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-06 11:42:26 [111,284,572ms] [Info] [omni.server.app] [4070] ../../../source/apps/omni.server/main.cpp:main():106: Got signal to exit the application. 2023-01-06 11:42:26 [111,284,572ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():391: OmniverseServer shutdown process starting. 2023-01-06 11:42:26 [111,284,572ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():392: Waiting for any outstanding transactions to complete. 2023-01-06 11:42:26 [111,284,572ms] [Info] [omni.server] [4070] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():396: Releasing WebSocket listen ports 2023-01-06 11:42:26 [111,284,572ms] [Info] [omni.server] [2395] ../../../source/plugins/omni.server/ApiConnection.cpp:shutdownCore():70: Shutdown Core 2023-01-06 11:43:29 [0ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-06 11:43:29 [4ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-06 11:43:29 [4ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-06 11:43:29 [9ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-06 11:43:29 [11ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-06 11:43:29 [11ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-06 11:43:29 [12ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-06 11:43:29 [13ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-06 11:43:29 [13ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-06 11:43:29 [16ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-06 11:43:29 [16ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-06 11:43:29 [18ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-06 11:43:29 [18ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-06 11:43:29 [19ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-06 11:43:29 [21ms] [Info] [carb] [3189] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-06 11:43:29 [21ms] [Info] [carb] [3189] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-06 11:43:29 [21ms] [Info] [carb] [3189] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-06 11:43:29 [21ms] [Info] [carb] [3189] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-06 11:43:29 [21ms] [Info] [carb] [3189] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-06 11:43:29 [21ms] [Info] [carb] [3189] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-06 11:43:29 [21ms] [Info] [omni.server] [3189] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-06 11:43:29 [21ms] [Info] [omni.server] [3189] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-06 11:43:29 [21ms] [Info] [omni.server] [3189] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-06 11:43:29 [22ms] [Info] [omni.server] [3189] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-06 11:43:29 [22ms] [Info] [omni.server] [3189] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-06 11:43:29 [22ms] [Info] [omni.server] [3189] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-06 11:43:29 [22ms] [Info] [omni.server] [3189] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-06 11:43:29 [22ms] [Info] [omni.server] [3189] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-06 11:43:29 [22ms] [Info] [omni.server] [3189] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-06 11:43:29 [22ms] [Info] [omni.server] [3189] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-06 11:43:29 [22ms] [Info] [carb] [3189] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-06 11:43:29 [23ms] [Info] [carb] [3189] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-06 11:43:29 [24ms] [Info] [carb] [3189] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-06 11:43:29 [25ms] [Info] [carb] [3189] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-06 11:43:29 [25ms] [Info] [carb.launcher.plugin] [3189] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x15264a0 for './omni-lft-service/omni-lft-service'. 2023-01-06 11:43:29 [25ms] [Info] [carb.launcher.plugin] [3189] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3189. 2023-01-06 11:43:29 [25ms] [Info] [carb] [3189] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-06 11:43:29 [25ms] [Info] [carb] [3189] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-06 11:43:29 [27ms] [Info] [carb.launcher.plugin] [3208] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3209 from the parent 3189. 2023-01-06 11:43:29 [27ms] [Info] [carb.launcher.plugin] [3208] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3209. 2023-01-06 11:43:29 [28ms] [Info] [carb.launcher.plugin] [3208] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3209. 2023-01-06 11:43:29 [29ms] [Info] [carb.launcher.plugin] [3189] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3209 from the parent 3189. 2023-01-06 11:43:29 [29ms] [Info] [carb.launcher.plugin] [3189] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3209 {process = 0x15264a0} 2023-01-06 11:43:29 [29ms] [Info] [carb.launcher.plugin] [3210] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3209. 2023-01-06 11:43:29 [29ms] [Info] [omni.core] [3189] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-06 11:43:29 [29ms] [Info] [omni.core] [3189] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-06 11:43:29 [29ms] [Info] [carb.launcher.plugin] [3211] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3209. 2023-01-06 11:43:29 [434ms] [Error] [omni.core] [3211] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-06 11:43:29 [441ms] [Error] [omni.core] [3211] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-06 20:43:29,536 - system_log - INFO - reconnecting ... 2023-01-06 11:43:29 [450ms] [Error] [omni.core] [3211] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-06 20:43:29,545 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 11:43:29 [0ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-06 11:43:29 [0ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-06 11:43:29 [0ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-06 11:43:29 [0ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-06 11:43:29 [0ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-06 11:43:29 [0ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-06 11:43:29 [0ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-06 11:43:29 [0ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-06 11:43:29 [0ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-06 11:43:29 [0ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-06 11:43:29 [1ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-06 11:43:29 [1ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-06 11:43:29 [1ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-06 11:43:29 [1ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-06 11:43:29 [1ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-06 11:43:29 [1ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-06 11:43:29 [1ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-06 11:43:29 [1ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-06 11:43:29 [1ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-06 11:43:29 [1ms] [Info] [omni.server] [3425] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-06 11:43:29 [1ms] [Info] [omni.server] [3425] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-06 11:43:29 [2ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-06 11:43:29 [2ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-06 11:43:29 [2ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-06 11:43:29 [2ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-06 11:43:29 [2ms] [Info] [carb.launcher.plugin] [3425] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xc4e0d0 for './omni-lft-service/omni-lft-service'. 2023-01-06 11:43:29 [2ms] [Info] [carb.launcher.plugin] [3425] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3425. 2023-01-06 11:43:29 [2ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-06 11:43:29 [2ms] [Info] [carb] [3425] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-06 11:43:29 [3ms] [Info] [carb.launcher.plugin] [3438] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3439 from the parent 3425. 2023-01-06 11:43:29 [3ms] [Info] [carb.launcher.plugin] [3438] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3439. 2023-01-06 11:43:29 [4ms] [Info] [carb.launcher.plugin] [3438] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3439. 2023-01-06 11:43:29 [4ms] [Info] [carb.launcher.plugin] [3425] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3439 from the parent 3425. 2023-01-06 11:43:29 [4ms] [Info] [carb.launcher.plugin] [3425] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3439 {process = 0xc4e0d0} 2023-01-06 11:43:29 [4ms] [Info] [omni.core] [3425] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-06 11:43:29 [4ms] [Info] [omni.core] [3425] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-06 11:43:29 [4ms] [Info] [carb.launcher.plugin] [3440] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3439. 2023-01-06 11:43:29 [4ms] [Info] [carb.launcher.plugin] [3441] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3439. 2023-01-06 11:43:30 [237ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-06 11:43:30 [240ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-06 20:43:30,185 - system_log - INFO - reconnecting ... 2023-01-06 11:43:30 [248ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-06 20:43:30,192 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 11:43:31 [1,249ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 20:43:31,193 - system_log - INFO - reconnecting ... 2023-01-06 11:43:31 [1,249ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-06 20:43:31,194 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 11:43:31 [2,004ms] [Info] [omni.core] [3431] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-06 11:43:31 [2,004ms] [Info] [omni.core] [3431] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-06 11:43:31 [2,005ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-06 11:43:31 [2,005ms] [Info] [omni.core] [3431] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-06 11:43:31 [2,008ms] [Info] [omni.core] [3425] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-06 11:43:31 [2,008ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-06 11:43:32 [2,086ms] [Warning] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-06 11:43:32 [2,086ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-06 11:43:32 [2,115ms] [Warning] [carb] [3425] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-06 11:43:32 [2,115ms] [Info] [carb] [3425] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-06 11:43:32 [2,115ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-01-06 11:43:32 [2,125ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-01-06 11:43:32 [2,125ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 17536 records 2023-01-06 11:43:32 [2,130ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1001 / 17536 records 2023-01-06 11:43:32 [2,134ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 2001 / 17536 records 2023-01-06 11:43:32 [2,137ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 3001 / 17536 records 2023-01-06 11:43:32 [2,139ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 4001 / 17536 records 2023-01-06 11:43:32 [2,142ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 5001 / 17536 records 2023-01-06 11:43:32 [2,145ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 6001 / 17536 records 2023-01-06 11:43:32 [2,146ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 7001 / 17536 records 2023-01-06 11:43:32 [2,147ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 8001 / 17536 records 2023-01-06 11:43:32 [2,147ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 9001 / 17536 records 2023-01-06 11:43:32 [2,147ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 10001 / 17536 records 2023-01-06 11:43:32 [2,148ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 11001 / 17536 records 2023-01-06 11:43:32 [2,148ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 12001 / 17536 records 2023-01-06 11:43:32 [2,148ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 13001 / 17536 records 2023-01-06 11:43:32 [2,149ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 14001 / 17536 records 2023-01-06 11:43:32 [2,149ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 15001 / 17536 records 2023-01-06 11:43:32 [2,150ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 16001 / 17536 records 2023-01-06 11:43:32 [2,152ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 17001 / 17536 records 2023-01-06 11:43:32 [2,152ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-01-06 11:43:32 [2,154ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 4242 records 2023-01-06 11:43:32 [2,156ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1001 / 4242 records 2023-01-06 11:43:32 [2,157ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 2001 / 4242 records 2023-01-06 11:43:32 [2,159ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 3001 / 4242 records 2023-01-06 11:43:32 [2,161ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 4001 / 4242 records 2023-01-06 11:43:32 [2,161ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-01-06 11:43:32 [2,168ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-01-06 11:43:32 [2,168ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-01-06 11:43:32 [2,168ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 17536 records 2023-01-06 11:43:32 [2,168ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1001 / 17536 records 2023-01-06 11:43:32 [2,171ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 2001 / 17536 records 2023-01-06 11:43:32 [2,174ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 3001 / 17536 records 2023-01-06 11:43:32 [2,178ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 4001 / 17536 records 2023-01-06 11:43:32 [2,182ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 5001 / 17536 records 2023-01-06 11:43:32 [2,186ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 6001 / 17536 records 2023-01-06 11:43:32 [2,192ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 7001 / 17536 records 2023-01-06 11:43:32 [2,199ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 8001 / 17536 records 2023-01-06 11:43:32 [2,206ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 9001 / 17536 records 2023-01-06 11:43:32 [2,213ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 10001 / 17536 records 2023-01-06 11:43:32 [2,221ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 11001 / 17536 records 2023-01-06 11:43:32 [2,228ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 12001 / 17536 records 2023-01-06 11:43:32 [2,235ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 13001 / 17536 records 2023-01-06 11:43:32 [2,242ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 14001 / 17536 records 2023-01-06 11:43:32 [2,249ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 15001 / 17536 records 2023-01-06 11:43:32 [2,250ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 20:43:32,195 - system_log - INFO - reconnecting ... 2023-01-06 11:43:32 [2,251ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-06 20:43:32,195 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 11:43:32 [2,254ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 16001 / 17536 records 2023-01-06 11:43:32 [2,260ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 17001 / 17536 records 2023-01-06 11:43:32 [2,264ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-01-06 11:43:32 [2,264ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-01-06 11:43:32 [2,264ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1 / 7697 records 2023-01-06 11:43:32 [2,265ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1001 / 7697 records 2023-01-06 11:43:32 [2,266ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 2001 / 7697 records 2023-01-06 11:43:32 [2,267ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 3001 / 7697 records 2023-01-06 11:43:32 [2,268ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 4001 / 7697 records 2023-01-06 11:43:32 [2,269ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 5001 / 7697 records 2023-01-06 11:43:32 [2,270ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 6001 / 7697 records 2023-01-06 11:43:32 [2,270ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 7001 / 7697 records 2023-01-06 11:43:32 [2,271ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-01-06 11:43:32 [2,271ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-01-06 11:43:32 [2,280ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-01-06 11:43:32 [2,280ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-01-06 11:43:32 [2,280ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-06 11:43:32 [2,280ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-06 11:43:32 [2,280ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-06 11:43:32 [2,280ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-06 11:43:32 [2,280ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-06 11:43:32 [2,280ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-06 11:43:32 [2,281ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-06 11:43:32 [2,281ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-06 11:43:32 [2,281ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-06 11:43:32 [2,281ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-06 11:43:32 [2,282ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-06 11:43:32 [2,282ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-06 11:43:32 [2,283ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-06 11:43:32 [2,283ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-06 11:43:32 [2,283ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-06 11:43:32 [2,283ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-06 11:43:32 [2,284ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-06 11:43:32 [2,291ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-06 11:43:32 [2,298ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-06 11:43:32 [2,304ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-06 11:43:32 [2,311ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-06 11:43:32 [2,317ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-06 11:43:32 [2,324ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-06 11:43:32 [2,331ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-06 11:43:32 [2,340ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-06 11:43:32 [2,348ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-06 11:43:32 [2,356ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-06 11:43:32 [2,363ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-06 11:43:32 [2,372ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-06 11:43:32 [2,380ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-06 11:43:32 [2,388ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-06 11:43:32 [2,396ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-06 11:43:32 [2,405ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-06 11:43:32 [2,411ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-06 11:43:32 [2,414ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-01-06 11:43:32 [2,440ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-01-06 11:43:32 [2,440ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-01-06 11:43:32 [2,446ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-01-06 11:43:32 [2,447ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-01-06 11:43:32 [2,447ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-01-06 11:43:32 [2,447ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-01-06 11:43:32 [2,447ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-01-06 11:43:32 [2,447ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-01-06 11:43:32 [2,447ms] [Info] [omni.meta-upgrade] [3425] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-01-06 11:43:32 [2,462ms] [Info] [omni.core] [3425] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-01-06 11:43:32 [2,465ms] [Info] [omni.core] [3425] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-01-06 11:43:32 [2,467ms] [Info] [omni.core] [3425] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-01-06 11:43:32 [2,467ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-01-06 11:43:32 [2,467ms] [Info] [omni.core] [3433] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-01-06 11:43:32 [2,468ms] [Info] [omni.core] [3430] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-01-06 11:43:32 [2,468ms] [Info] [carb.launcher.plugin] [3430] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7f674c001bb0 for './omni.delta-applier.app'. 2023-01-06 11:43:32 [2,468ms] [Info] [carb.launcher.plugin] [3430] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 3425. 2023-01-06 11:43:32 [2,468ms] [Info] [carb.launcher.plugin] [3438] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4007 from the parent 3425. 2023-01-06 11:43:32 [2,468ms] [Info] [carb.launcher.plugin] [3438] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4007. 2023-01-06 11:43:32 [2,469ms] [Info] [carb.launcher.plugin] [3438] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4007. 2023-01-06 11:43:32 [2,469ms] [Info] [carb.launcher.plugin] [3430] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4007 from the parent 3425. 2023-01-06 11:43:32 [2,469ms] [Info] [carb.launcher.plugin] [3430] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4007 {process = 0x7f674c001bb0} 2023-01-06 11:43:32 [2,469ms] [Info] [carb.launcher.plugin] [4008] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4007. 2023-01-06 11:43:32 [2,469ms] [Info] [carb.launcher.plugin] [4009] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4007. 2023-01-06 11:43:32 [2,671ms] [Info] [omni.core] [3432] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-01-06 11:43:33 [3,251ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 20:43:33,196 - system_log - INFO - reconnecting ... 2023-01-06 11:43:33 [3,252ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-06 20:43:33,196 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 11:43:34 [4,253ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 20:43:34,197 - system_log - INFO - reconnecting ... 2023-01-06 11:43:34 [4,253ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-06 20:43:34,198 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 11:43:34 [4,467ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-01-06 11:43:34 [4,477ms] [Info] [omni.server] [3425] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-01-06 11:43:34 [4,477ms] [Info] [omni.signalhandler] [3425] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-01-06 11:43:34 [4,477ms] [Info] [omni.signalhandler] [3425] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-01-06 11:43:34 [4,477ms] [Info] [omni.signalhandler] [3425] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-01-06 11:43:35 [5,254ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-06 20:43:35,199 - system_log - INFO - reconnecting ... 2023-01-06 11:43:35 [5,256ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-06 20:43:35,200 - system_log - INFO - connected 2023-01-06 11:43:35 [5,257ms] [Error] [omni.core] [3441] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-01-09 06:55:55 [0ms] [Info] [carb] [3094] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-09 06:55:55 [1ms] [Info] [carb] [3094] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-09 06:55:55 [0ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-09 06:55:55 [0ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-09 06:55:55 [0ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-09 06:55:55 [1ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-09 06:55:55 [5ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-09 06:55:55 [7ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-09 06:55:55 [8ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-09 06:55:55 [9ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-09 06:55:55 [9ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-09 06:55:55 [10ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-09 06:55:55 [12ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-09 06:55:55 [12ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-09 06:55:55 [14ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-09 06:55:55 [14ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-09 06:55:55 [15ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-09 06:55:55 [17ms] [Info] [carb] [3113] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-09 06:55:55 [17ms] [Info] [carb] [3113] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-09 06:55:55 [17ms] [Info] [carb] [3113] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-09 06:55:55 [17ms] [Info] [carb] [3113] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-09 06:55:55 [17ms] [Info] [carb] [3113] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-09 06:55:55 [17ms] [Info] [carb] [3113] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-09 06:55:55 [18ms] [Info] [omni.server] [3113] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-09 06:55:55 [18ms] [Info] [omni.server] [3113] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-09 06:55:55 [18ms] [Info] [omni.server] [3113] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-09 06:55:55 [19ms] [Info] [omni.server] [3113] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-09 06:55:55 [19ms] [Info] [omni.server] [3113] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-09 06:55:55 [19ms] [Info] [omni.server] [3113] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-09 06:55:55 [19ms] [Info] [omni.server] [3113] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-09 06:55:55 [19ms] [Info] [omni.server] [3113] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-09 06:55:55 [19ms] [Info] [omni.server] [3113] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-09 06:55:55 [19ms] [Info] [omni.server] [3113] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-09 06:55:55 [19ms] [Info] [carb] [3113] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-09 06:55:55 [20ms] [Info] [carb] [3113] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-09 06:55:55 [21ms] [Info] [carb] [3113] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-09 06:55:55 [22ms] [Info] [carb] [3113] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-09 06:55:55 [22ms] [Info] [carb.launcher.plugin] [3113] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x25424b0 for './omni-lft-service/omni-lft-service'. 2023-01-09 06:55:55 [22ms] [Info] [carb.launcher.plugin] [3113] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3113. 2023-01-09 06:55:55 [22ms] [Info] [carb] [3113] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-09 06:55:55 [22ms] [Info] [carb] [3113] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-09 06:55:55 [23ms] [Info] [carb.launcher.plugin] [3131] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3132 from the parent 3113. 2023-01-09 06:55:55 [23ms] [Info] [carb.launcher.plugin] [3131] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3132. 2023-01-09 06:55:55 [24ms] [Info] [carb.launcher.plugin] [3131] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3132. 2023-01-09 06:55:55 [24ms] [Info] [carb.launcher.plugin] [3113] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3132 from the parent 3113. 2023-01-09 06:55:55 [24ms] [Info] [carb.launcher.plugin] [3113] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3132 {process = 0x25424b0} 2023-01-09 06:55:55 [24ms] [Info] [omni.core] [3113] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-09 06:55:55 [24ms] [Info] [omni.core] [3113] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-09 06:55:55 [24ms] [Info] [carb.launcher.plugin] [3133] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3132. 2023-01-09 06:55:55 [24ms] [Info] [carb.launcher.plugin] [3134] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3132. 2023-01-09 06:55:56 [410ms] [Error] [omni.core] [3134] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-09 06:55:56 [416ms] [Error] [omni.core] [3134] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-09 15:55:56,302 - system_log - INFO - reconnecting ... 2023-01-09 06:55:56 [424ms] [Error] [omni.core] [3134] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-09 15:55:56,310 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-09 06:55:59 [0ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-09 06:55:59 [1ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-09 06:55:59 [1ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-09 06:55:59 [1ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-09 06:55:59 [1ms] [Info] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-09 06:55:59 [1ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-09 06:55:59 [1ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-09 06:55:59 [1ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-09 06:55:59 [1ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-09 06:55:59 [1ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-09 06:55:59 [1ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-09 06:55:59 [1ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-09 06:55:59 [1ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-09 06:55:59 [1ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-09 06:55:59 [1ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-09 06:55:59 [1ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-09 06:55:59 [1ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-09 06:55:59 [1ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-09 06:55:59 [1ms] [Info] [omni.server] [4005] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-09 06:55:59 [2ms] [Info] [omni.server] [4005] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-09 06:55:59 [2ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-09 06:55:59 [2ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-09 06:55:59 [2ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-09 06:55:59 [2ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-09 06:55:59 [2ms] [Info] [carb.launcher.plugin] [4005] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x10480c0 for './omni-lft-service/omni-lft-service'. 2023-01-09 06:55:59 [2ms] [Info] [carb.launcher.plugin] [4005] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 4005. 2023-01-09 06:55:59 [2ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-09 06:55:59 [2ms] [Info] [carb] [4005] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-09 06:55:59 [3ms] [Info] [carb.launcher.plugin] [4018] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4019 from the parent 4005. 2023-01-09 06:55:59 [3ms] [Info] [carb.launcher.plugin] [4018] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4019. 2023-01-09 06:55:59 [3ms] [Info] [carb.launcher.plugin] [4018] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4019. 2023-01-09 06:55:59 [3ms] [Info] [carb.launcher.plugin] [4005] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4019 from the parent 4005. 2023-01-09 06:55:59 [3ms] [Info] [carb.launcher.plugin] [4005] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4019 {process = 0x10480c0} 2023-01-09 06:55:59 [3ms] [Info] [omni.core] [4005] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-09 06:55:59 [3ms] [Info] [omni.core] [4005] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-09 06:55:59 [3ms] [Info] [carb.launcher.plugin] [4020] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4019. 2023-01-09 06:55:59 [3ms] [Info] [carb.launcher.plugin] [4021] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4019. 2023-01-09 06:56:00 [228ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-09 06:56:00 [231ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-09 15:56:00,016 - system_log - INFO - reconnecting ... 2023-01-09 06:56:00 [238ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-09 15:56:00,023 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 06:56:01 [1,240ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 15:56:01,025 - system_log - INFO - reconnecting ... 2023-01-09 06:56:01 [1,240ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-09 15:56:01,025 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 06:56:01 [2,004ms] [Info] [omni.core] [4010] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-09 06:56:01 [2,004ms] [Info] [omni.core] [4010] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-09 06:56:01 [2,006ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-09 06:56:01 [2,006ms] [Info] [omni.core] [4010] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-09 06:56:01 [2,012ms] [Info] [omni.core] [4005] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-09 06:56:01 [2,012ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-09 06:56:01 [2,096ms] [Warning] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-09 06:56:01 [2,096ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-09 06:56:01 [2,127ms] [Warning] [carb] [4005] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-09 06:56:01 [2,127ms] [Info] [carb] [4005] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-09 06:56:01 [2,127ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-01-09 06:56:01 [2,142ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-01-09 06:56:01 [2,142ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 17536 records 2023-01-09 06:56:01 [2,147ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1001 / 17536 records 2023-01-09 06:56:01 [2,150ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 2001 / 17536 records 2023-01-09 06:56:01 [2,153ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 3001 / 17536 records 2023-01-09 06:56:01 [2,155ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 4001 / 17536 records 2023-01-09 06:56:01 [2,157ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 5001 / 17536 records 2023-01-09 06:56:01 [2,160ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 6001 / 17536 records 2023-01-09 06:56:01 [2,161ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 7001 / 17536 records 2023-01-09 06:56:01 [2,162ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 8001 / 17536 records 2023-01-09 06:56:01 [2,162ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 9001 / 17536 records 2023-01-09 06:56:01 [2,162ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 10001 / 17536 records 2023-01-09 06:56:01 [2,163ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 11001 / 17536 records 2023-01-09 06:56:01 [2,163ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 12001 / 17536 records 2023-01-09 06:56:01 [2,163ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 13001 / 17536 records 2023-01-09 06:56:01 [2,164ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 14001 / 17536 records 2023-01-09 06:56:01 [2,164ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 15001 / 17536 records 2023-01-09 06:56:01 [2,165ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 16001 / 17536 records 2023-01-09 06:56:01 [2,166ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 17001 / 17536 records 2023-01-09 06:56:01 [2,167ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-01-09 06:56:01 [2,168ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 4242 records 2023-01-09 06:56:01 [2,170ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1001 / 4242 records 2023-01-09 06:56:01 [2,172ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 2001 / 4242 records 2023-01-09 06:56:01 [2,174ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 3001 / 4242 records 2023-01-09 06:56:01 [2,176ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 4001 / 4242 records 2023-01-09 06:56:01 [2,176ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-01-09 06:56:01 [2,182ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-01-09 06:56:01 [2,182ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-01-09 06:56:01 [2,182ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 17536 records 2023-01-09 06:56:01 [2,182ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1001 / 17536 records 2023-01-09 06:56:01 [2,185ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 2001 / 17536 records 2023-01-09 06:56:01 [2,188ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 3001 / 17536 records 2023-01-09 06:56:01 [2,192ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 4001 / 17536 records 2023-01-09 06:56:01 [2,195ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 5001 / 17536 records 2023-01-09 06:56:01 [2,199ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 6001 / 17536 records 2023-01-09 06:56:01 [2,205ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 7001 / 17536 records 2023-01-09 06:56:01 [2,212ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 8001 / 17536 records 2023-01-09 06:56:02 [2,219ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 9001 / 17536 records 2023-01-09 06:56:02 [2,226ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 10001 / 17536 records 2023-01-09 06:56:02 [2,233ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 11001 / 17536 records 2023-01-09 06:56:02 [2,239ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 12001 / 17536 records 2023-01-09 06:56:02 [2,241ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 15:56:02,026 - system_log - INFO - reconnecting ... 2023-01-09 06:56:02 [2,242ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-09 15:56:02,027 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 06:56:02 [2,246ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 13001 / 17536 records 2023-01-09 06:56:02 [2,253ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 14001 / 17536 records 2023-01-09 06:56:02 [2,260ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 15001 / 17536 records 2023-01-09 06:56:02 [2,265ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 16001 / 17536 records 2023-01-09 06:56:02 [2,271ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 17001 / 17536 records 2023-01-09 06:56:02 [2,274ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-01-09 06:56:02 [2,274ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-01-09 06:56:02 [2,274ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1 / 7697 records 2023-01-09 06:56:02 [2,275ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1001 / 7697 records 2023-01-09 06:56:02 [2,276ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 2001 / 7697 records 2023-01-09 06:56:02 [2,277ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 3001 / 7697 records 2023-01-09 06:56:02 [2,278ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 4001 / 7697 records 2023-01-09 06:56:02 [2,279ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 5001 / 7697 records 2023-01-09 06:56:02 [2,279ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 6001 / 7697 records 2023-01-09 06:56:02 [2,280ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 7001 / 7697 records 2023-01-09 06:56:02 [2,281ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-01-09 06:56:02 [2,281ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-01-09 06:56:02 [2,289ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-01-09 06:56:02 [2,289ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-01-09 06:56:02 [2,289ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-09 06:56:02 [2,289ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-09 06:56:02 [2,289ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-09 06:56:02 [2,289ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-09 06:56:02 [2,289ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-09 06:56:02 [2,289ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-09 06:56:02 [2,289ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-09 06:56:02 [2,290ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-09 06:56:02 [2,290ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-09 06:56:02 [2,290ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-09 06:56:02 [2,291ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-09 06:56:02 [2,291ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-09 06:56:02 [2,291ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-09 06:56:02 [2,292ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-09 06:56:02 [2,292ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-09 06:56:02 [2,292ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-09 06:56:02 [2,293ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-09 06:56:02 [2,300ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17536 records 2023-01-09 06:56:02 [2,306ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17536 records 2023-01-09 06:56:02 [2,313ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17536 records 2023-01-09 06:56:02 [2,319ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17536 records 2023-01-09 06:56:02 [2,326ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17536 records 2023-01-09 06:56:02 [2,332ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17536 records 2023-01-09 06:56:02 [2,341ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17536 records 2023-01-09 06:56:02 [2,350ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17536 records 2023-01-09 06:56:02 [2,359ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17536 records 2023-01-09 06:56:02 [2,368ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17536 records 2023-01-09 06:56:02 [2,375ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17536 records 2023-01-09 06:56:02 [2,385ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17536 records 2023-01-09 06:56:02 [2,394ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17536 records 2023-01-09 06:56:02 [2,403ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17536 records 2023-01-09 06:56:02 [2,412ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17536 records 2023-01-09 06:56:02 [2,421ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17536 records 2023-01-09 06:56:02 [2,428ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17536 records 2023-01-09 06:56:02 [2,431ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-01-09 06:56:02 [2,461ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-01-09 06:56:02 [2,461ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-01-09 06:56:02 [2,468ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-01-09 06:56:02 [2,468ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-01-09 06:56:02 [2,468ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-01-09 06:56:02 [2,468ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-01-09 06:56:02 [2,468ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-01-09 06:56:02 [2,469ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-01-09 06:56:02 [2,469ms] [Info] [omni.meta-upgrade] [4005] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-01-09 06:56:02 [2,486ms] [Info] [omni.core] [4005] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-01-09 06:56:02 [2,489ms] [Info] [omni.core] [4005] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-01-09 06:56:02 [2,491ms] [Info] [omni.core] [4005] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-01-09 06:56:02 [2,491ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-01-09 06:56:02 [2,491ms] [Info] [omni.core] [4012] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-01-09 06:56:02 [2,491ms] [Info] [omni.core] [4013] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-01-09 06:56:02 [2,491ms] [Info] [carb.launcher.plugin] [4013] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7fee90003350 for './omni.delta-applier.app'. 2023-01-09 06:56:02 [2,491ms] [Info] [carb.launcher.plugin] [4013] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 4005. 2023-01-09 06:56:02 [2,492ms] [Info] [carb.launcher.plugin] [4018] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4082 from the parent 4005. 2023-01-09 06:56:02 [2,492ms] [Info] [carb.launcher.plugin] [4018] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4082. 2023-01-09 06:56:02 [2,493ms] [Info] [carb.launcher.plugin] [4018] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4082. 2023-01-09 06:56:02 [2,493ms] [Info] [carb.launcher.plugin] [4013] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4082 from the parent 4005. 2023-01-09 06:56:02 [2,493ms] [Info] [carb.launcher.plugin] [4013] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4082 {process = 0x7fee90003350} 2023-01-09 06:56:02 [2,493ms] [Info] [carb.launcher.plugin] [4083] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4082. 2023-01-09 06:56:02 [2,493ms] [Info] [carb.launcher.plugin] [4084] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4082. 2023-01-09 06:56:02 [2,696ms] [Info] [omni.core] [4010] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-01-09 06:56:03 [3,244ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 15:56:03,028 - system_log - INFO - reconnecting ... 2023-01-09 06:56:03 [3,246ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-09 15:56:03,031 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 06:56:04 [4,248ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 15:56:04,032 - system_log - INFO - reconnecting ... 2023-01-09 06:56:04 [4,250ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-09 15:56:04,035 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 06:56:04 [4,491ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-01-09 06:56:04 [4,504ms] [Info] [omni.server] [4005] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-01-09 06:56:04 [4,504ms] [Info] [omni.signalhandler] [4005] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-01-09 06:56:04 [4,504ms] [Info] [omni.signalhandler] [4005] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-01-09 06:56:04 [4,504ms] [Info] [omni.signalhandler] [4005] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-01-09 06:56:05 [5,251ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-09 15:56:05,036 - system_log - INFO - reconnecting ... 2023-01-09 06:56:05 [5,258ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-09 15:56:05,043 - system_log - INFO - connected 2023-01-09 06:56:05 [5,259ms] [Error] [omni.core] [4021] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-01-09 10:41:09 [13,509,543ms] [Warning] [carb] [4013] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::ICore v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-10 06:11:58 [0ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-10 06:11:58 [1ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-10 06:11:58 [4ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-10 06:11:58 [5ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-10 06:11:58 [9ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-10 06:11:58 [11ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-10 06:11:58 [12ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-10 06:11:58 [13ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-10 06:11:58 [13ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-10 06:11:58 [14ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-10 06:11:58 [16ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-10 06:11:58 [16ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-10 06:11:58 [18ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-10 06:11:58 [19ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-10 06:11:58 [20ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-10 06:11:58 [21ms] [Info] [carb] [3084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-10 06:11:58 [21ms] [Info] [carb] [3084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-10 06:11:58 [21ms] [Info] [carb] [3084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-10 06:11:58 [21ms] [Info] [carb] [3084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-10 06:11:58 [21ms] [Info] [carb] [3084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-10 06:11:58 [21ms] [Info] [carb] [3084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-10 06:11:58 [22ms] [Info] [omni.server] [3084] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-10 06:11:58 [22ms] [Info] [omni.server] [3084] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-10 06:11:58 [22ms] [Info] [omni.server] [3084] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-10 06:11:58 [22ms] [Info] [omni.server] [3084] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-10 06:11:58 [22ms] [Info] [omni.server] [3084] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-10 06:11:58 [22ms] [Info] [omni.server] [3084] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-10 06:11:58 [22ms] [Info] [omni.server] [3084] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-10 06:11:58 [22ms] [Info] [omni.server] [3084] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-10 06:11:58 [22ms] [Info] [omni.server] [3084] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-10 06:11:58 [23ms] [Info] [omni.server] [3084] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-10 06:11:58 [23ms] [Info] [carb] [3084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-10 06:11:58 [24ms] [Info] [carb] [3084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-10 06:11:58 [24ms] [Info] [carb] [3084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-10 06:11:58 [26ms] [Info] [carb] [3084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-10 06:11:58 [26ms] [Info] [carb.launcher.plugin] [3084] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x13674a0 for './omni-lft-service/omni-lft-service'. 2023-01-10 06:11:58 [26ms] [Info] [carb.launcher.plugin] [3084] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3084. 2023-01-10 06:11:58 [26ms] [Info] [carb] [3084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-10 06:11:58 [26ms] [Info] [carb] [3084] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-10 06:11:58 [26ms] [Info] [carb.launcher.plugin] [3103] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3104 from the parent 3084. 2023-01-10 06:11:58 [26ms] [Info] [carb.launcher.plugin] [3103] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3104. 2023-01-10 06:11:58 [28ms] [Info] [carb.launcher.plugin] [3103] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3104. 2023-01-10 06:11:58 [28ms] [Info] [carb.launcher.plugin] [3084] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3104 from the parent 3084. 2023-01-10 06:11:58 [28ms] [Info] [carb.launcher.plugin] [3084] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3104 {process = 0x13674a0} 2023-01-10 06:11:58 [28ms] [Info] [carb.launcher.plugin] [3105] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3104. 2023-01-10 06:11:58 [28ms] [Info] [omni.core] [3084] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-10 06:11:58 [28ms] [Info] [omni.core] [3084] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-10 06:11:58 [29ms] [Info] [carb.launcher.plugin] [3106] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3104. 2023-01-10 06:11:59 [395ms] [Error] [omni.core] [3106] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-10 06:11:59 [410ms] [Error] [omni.core] [3106] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-10 15:11:59,134 - system_log - INFO - reconnecting ... 2023-01-10 06:11:59 [419ms] [Error] [omni.core] [3106] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-10 15:11:59,143 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-10 06:12:00 [0ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-10 06:12:00 [1ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-10 06:12:00 [1ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-10 06:12:00 [1ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-10 06:12:00 [1ms] [Info] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-10 06:12:00 [1ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-10 06:12:00 [1ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-10 06:12:00 [1ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-10 06:12:00 [1ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-10 06:12:00 [1ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-10 06:12:00 [1ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-10 06:12:00 [1ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-10 06:12:00 [1ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-10 06:12:00 [1ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-10 06:12:00 [1ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-10 06:12:00 [1ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-10 06:12:00 [1ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-10 06:12:00 [1ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-10 06:12:00 [1ms] [Info] [omni.server] [3377] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-10 06:12:00 [1ms] [Info] [omni.server] [3377] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-10 06:12:00 [1ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-10 06:12:00 [2ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-10 06:12:00 [2ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-10 06:12:00 [2ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-10 06:12:00 [2ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xb8e0c0 for './omni-lft-service/omni-lft-service'. 2023-01-10 06:12:00 [2ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 3377. 2023-01-10 06:12:00 [2ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-10 06:12:00 [2ms] [Info] [carb] [3377] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-10 06:12:00 [2ms] [Info] [carb.launcher.plugin] [3391] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3392 from the parent 3377. 2023-01-10 06:12:00 [3ms] [Info] [carb.launcher.plugin] [3391] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3392. 2023-01-10 06:12:00 [3ms] [Info] [carb.launcher.plugin] [3391] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3392. 2023-01-10 06:12:00 [3ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3392 from the parent 3377. 2023-01-10 06:12:00 [3ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3392 {process = 0xb8e0c0} 2023-01-10 06:12:00 [3ms] [Info] [omni.core] [3377] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-10 06:12:00 [3ms] [Info] [omni.core] [3377] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-10 06:12:00 [3ms] [Info] [carb.launcher.plugin] [3393] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3392. 2023-01-10 06:12:00 [3ms] [Info] [carb.launcher.plugin] [3394] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3392. 2023-01-10 06:12:00 [232ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-10 06:12:00 [236ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-10 15:12:00,550 - system_log - INFO - reconnecting ... 2023-01-10 06:12:00 [243ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-10 15:12:00,558 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 06:12:01 [1,245ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 15:12:01,560 - system_log - INFO - reconnecting ... 2023-01-10 06:12:01 [1,245ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-10 15:12:01,560 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 06:12:02 [2,003ms] [Info] [omni.core] [3385] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-10 06:12:02 [2,003ms] [Info] [omni.core] [3385] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-10 06:12:02 [2,004ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-10 06:12:02 [2,004ms] [Info] [omni.core] [3385] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-10 06:12:02 [2,008ms] [Info] [omni.core] [3377] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-10 06:12:02 [2,008ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-10 06:12:02 [2,096ms] [Warning] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-10 06:12:02 [2,096ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-10 06:12:02 [2,125ms] [Warning] [carb] [3377] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-10 06:12:02 [2,125ms] [Info] [carb] [3377] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-10 06:12:02 [2,125ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-01-10 06:12:02 [2,135ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-01-10 06:12:02 [2,135ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 17540 records 2023-01-10 06:12:02 [2,140ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1001 / 17540 records 2023-01-10 06:12:02 [2,143ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 2001 / 17540 records 2023-01-10 06:12:02 [2,145ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 3001 / 17540 records 2023-01-10 06:12:02 [2,147ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 4001 / 17540 records 2023-01-10 06:12:02 [2,150ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 5001 / 17540 records 2023-01-10 06:12:02 [2,152ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 6001 / 17540 records 2023-01-10 06:12:02 [2,154ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 7001 / 17540 records 2023-01-10 06:12:02 [2,154ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 8001 / 17540 records 2023-01-10 06:12:02 [2,155ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 9001 / 17540 records 2023-01-10 06:12:02 [2,155ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 10001 / 17540 records 2023-01-10 06:12:02 [2,156ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 11001 / 17540 records 2023-01-10 06:12:02 [2,156ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 12001 / 17540 records 2023-01-10 06:12:02 [2,156ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 13001 / 17540 records 2023-01-10 06:12:02 [2,156ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 14001 / 17540 records 2023-01-10 06:12:02 [2,157ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 15001 / 17540 records 2023-01-10 06:12:02 [2,158ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 16001 / 17540 records 2023-01-10 06:12:02 [2,159ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 17001 / 17540 records 2023-01-10 06:12:02 [2,159ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-01-10 06:12:02 [2,161ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 4243 records 2023-01-10 06:12:02 [2,163ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1001 / 4243 records 2023-01-10 06:12:02 [2,164ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 2001 / 4243 records 2023-01-10 06:12:02 [2,166ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 3001 / 4243 records 2023-01-10 06:12:02 [2,168ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 4001 / 4243 records 2023-01-10 06:12:02 [2,168ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-01-10 06:12:02 [2,174ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-01-10 06:12:02 [2,174ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-01-10 06:12:02 [2,174ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 17540 records 2023-01-10 06:12:02 [2,174ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1001 / 17540 records 2023-01-10 06:12:02 [2,177ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 2001 / 17540 records 2023-01-10 06:12:02 [2,180ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 3001 / 17540 records 2023-01-10 06:12:02 [2,184ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 4001 / 17540 records 2023-01-10 06:12:02 [2,187ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 5001 / 17540 records 2023-01-10 06:12:02 [2,191ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 6001 / 17540 records 2023-01-10 06:12:02 [2,196ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 7001 / 17540 records 2023-01-10 06:12:02 [2,203ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 8001 / 17540 records 2023-01-10 06:12:02 [2,209ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 9001 / 17540 records 2023-01-10 06:12:02 [2,216ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 10001 / 17540 records 2023-01-10 06:12:02 [2,223ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 11001 / 17540 records 2023-01-10 06:12:02 [2,230ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 12001 / 17540 records 2023-01-10 06:12:02 [2,236ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 13001 / 17540 records 2023-01-10 06:12:02 [2,243ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 14001 / 17540 records 2023-01-10 06:12:02 [2,247ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 15:12:02,561 - system_log - INFO - reconnecting ... 2023-01-10 06:12:02 [2,247ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-10 15:12:02,562 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 06:12:02 [2,249ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 15001 / 17540 records 2023-01-10 06:12:02 [2,255ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 16001 / 17540 records 2023-01-10 06:12:02 [2,261ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 17001 / 17540 records 2023-01-10 06:12:02 [2,264ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-01-10 06:12:02 [2,264ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-01-10 06:12:02 [2,264ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1 / 7698 records 2023-01-10 06:12:02 [2,265ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1001 / 7698 records 2023-01-10 06:12:02 [2,266ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 2001 / 7698 records 2023-01-10 06:12:02 [2,267ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 3001 / 7698 records 2023-01-10 06:12:02 [2,268ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 4001 / 7698 records 2023-01-10 06:12:02 [2,269ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 5001 / 7698 records 2023-01-10 06:12:02 [2,270ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 6001 / 7698 records 2023-01-10 06:12:02 [2,270ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 7001 / 7698 records 2023-01-10 06:12:02 [2,271ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-01-10 06:12:02 [2,271ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-01-10 06:12:02 [2,279ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-01-10 06:12:02 [2,279ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-01-10 06:12:02 [2,279ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17540 records 2023-01-10 06:12:02 [2,279ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17540 records 2023-01-10 06:12:02 [2,279ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17540 records 2023-01-10 06:12:02 [2,279ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17540 records 2023-01-10 06:12:02 [2,279ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17540 records 2023-01-10 06:12:02 [2,280ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17540 records 2023-01-10 06:12:02 [2,280ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17540 records 2023-01-10 06:12:02 [2,280ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17540 records 2023-01-10 06:12:02 [2,280ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17540 records 2023-01-10 06:12:02 [2,280ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17540 records 2023-01-10 06:12:02 [2,282ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17540 records 2023-01-10 06:12:02 [2,282ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17540 records 2023-01-10 06:12:02 [2,282ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17540 records 2023-01-10 06:12:02 [2,282ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17540 records 2023-01-10 06:12:02 [2,282ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17540 records 2023-01-10 06:12:02 [2,282ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17540 records 2023-01-10 06:12:02 [2,284ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17540 records 2023-01-10 06:12:02 [2,290ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17540 records 2023-01-10 06:12:02 [2,296ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17540 records 2023-01-10 06:12:02 [2,303ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17540 records 2023-01-10 06:12:02 [2,309ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17540 records 2023-01-10 06:12:02 [2,316ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17540 records 2023-01-10 06:12:02 [2,322ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17540 records 2023-01-10 06:12:02 [2,330ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17540 records 2023-01-10 06:12:02 [2,339ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17540 records 2023-01-10 06:12:02 [2,347ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17540 records 2023-01-10 06:12:02 [2,356ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17540 records 2023-01-10 06:12:02 [2,364ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17540 records 2023-01-10 06:12:02 [2,374ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17540 records 2023-01-10 06:12:02 [2,382ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17540 records 2023-01-10 06:12:02 [2,391ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17540 records 2023-01-10 06:12:02 [2,399ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17540 records 2023-01-10 06:12:02 [2,409ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17540 records 2023-01-10 06:12:02 [2,416ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17540 records 2023-01-10 06:12:02 [2,418ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-01-10 06:12:02 [2,447ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-01-10 06:12:02 [2,447ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-01-10 06:12:02 [2,453ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-01-10 06:12:02 [2,453ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-01-10 06:12:02 [2,453ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-01-10 06:12:02 [2,453ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-01-10 06:12:02 [2,453ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-01-10 06:12:02 [2,454ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-01-10 06:12:02 [2,454ms] [Info] [omni.meta-upgrade] [3377] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-01-10 06:12:02 [2,468ms] [Info] [omni.core] [3377] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-01-10 06:12:02 [2,470ms] [Info] [omni.core] [3377] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-01-10 06:12:02 [2,472ms] [Info] [omni.core] [3377] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-01-10 06:12:02 [2,472ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-01-10 06:12:02 [2,472ms] [Info] [omni.core] [3385] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-01-10 06:12:02 [2,473ms] [Info] [omni.core] [3383] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-01-10 06:12:02 [2,473ms] [Info] [carb.launcher.plugin] [3383] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7f2af0003e00 for './omni.delta-applier.app'. 2023-01-10 06:12:02 [2,473ms] [Info] [carb.launcher.plugin] [3383] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 3377. 2023-01-10 06:12:02 [2,473ms] [Info] [carb.launcher.plugin] [3391] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 3940 from the parent 3377. 2023-01-10 06:12:02 [2,473ms] [Info] [carb.launcher.plugin] [3391] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 3940. 2023-01-10 06:12:02 [2,474ms] [Info] [carb.launcher.plugin] [3391] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 3940. 2023-01-10 06:12:02 [2,475ms] [Info] [carb.launcher.plugin] [3383] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 3940 from the parent 3377. 2023-01-10 06:12:02 [2,475ms] [Info] [carb.launcher.plugin] [3383] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 3940 {process = 0x7f2af0003e00} 2023-01-10 06:12:02 [2,475ms] [Info] [carb.launcher.plugin] [3941] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 3940. 2023-01-10 06:12:02 [2,475ms] [Info] [carb.launcher.plugin] [3942] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 3940. 2023-01-10 06:12:02 [2,676ms] [Info] [omni.core] [3384] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-01-10 06:12:03 [3,248ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 15:12:03,563 - system_log - INFO - reconnecting ... 2023-01-10 06:12:03 [3,249ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-10 15:12:03,564 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 06:12:04 [4,250ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 15:12:04,565 - system_log - INFO - reconnecting ... 2023-01-10 06:12:04 [4,252ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-10 15:12:04,567 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 06:12:04 [4,473ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-01-10 06:12:04 [4,475ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-01-10 06:12:04 [4,475ms] [Info] [omni.signalhandler] [3377] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-01-10 06:12:04 [4,475ms] [Info] [omni.signalhandler] [3377] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-01-10 06:12:04 [4,475ms] [Info] [omni.signalhandler] [3377] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-01-10 06:12:05 [5,254ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "idl/connection/transport/omni.py", line 110, in prepare File "websockets/legacy/client.py", line 650, in __await_impl_timeout__ File "asyncio/tasks.py", line 494, in wait_for File "websockets/legacy/client.py", line 654, in __await_impl__ File "asyncio/base_events.py", line 1025, in create_connection File "asyncio/base_events.py", line 1010, in create_connection File "asyncio/base_events.py", line 924, in _connect_sock File "asyncio/selector_events.py", line 496, in sock_connect File "asyncio/selector_events.py", line 528, in _sock_connect_cb ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lft/server.py", line 560, in await_connection File "idl/connection/transport/omni.py", line 114, in prepare idl.connection.transport.TransportError: [Errno 111] Connect call failed ('127.0.0.1', 3007) 2023-01-10 15:12:05,569 - system_log - INFO - reconnecting ... 2023-01-10 06:12:05 [5,264ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-10 15:12:05,578 - system_log - INFO - connected 2023-01-10 06:12:05 [5,265ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-01-10 06:38:26 [1,585,700ms] [Warning] [carb] [3383] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::ICore v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-10 06:44:11 [1,931,666ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "lft/middlewares.py", line 98, in log_errors_middleware File "aiohttp/web_urldispatcher.py", line 197, in handler_wrapper File "lft/paths/download.py", line 65, in download_asset File "lft/paths/download.py", line 57, in download_asset_base File "aiohttp/web_response.py", line 514, in write File "aiohttp/http_writer.py", line 116, in write File "aiohttp/http_writer.py", line 76, in _write ConnectionResetError: Cannot write to closing transport Error handling request Traceback (most recent call last): File "aiohttp/web_protocol.py", line 435, in _handle_request File "aiohttp/web_app.py", line 504, in _handle File "aiohttp/web_middlewares.py", line 117, in impl File "lft/middlewares.py", line 66, in cors_middleware File "lft/middlewares.py", line 129, in verbose_access_log_middleware File "lft/middlewares.py", line 78, in filesystem_error_middleware File "lft/middlewares.py", line 98, in log_errors_middleware File "aiohttp/web_urldispatcher.py", line 197, in handler_wrapper File "lft/paths/download.py", line 65, in download_asset File "lft/paths/download.py", line 57, in download_asset_base File "aiohttp/web_response.py", line 514, in write File "aiohttp/http_writer.py", line 116, in write File "aiohttp/http_writer.py", line 76, in _write ConnectionResetError: Cannot write to closing transport 2023-01-11 01:29:36 [0ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:29:36 [0ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:29:36 [0ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:29:36 [0ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:29:36 [0ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:29:36 [0ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:29:36 [0ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:29:36 [0ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:29:36 [0ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:29:36 [0ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:29:36 [1ms] [Info] [carb] [28868] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:29:36 [1ms] [Info] [omni.server] [28868] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:29:36 [1ms] [Info] [omni.server] [28868] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:29:36 [1ms] [Info] [omni.server] [28868] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:29:36 [1ms] [Info] [omni.server] [28868] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:29:36 [1ms] [Info] [omni.server] [28868] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:29:36 [1ms] [Info] [omni.server] [28868] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:29:36 [1ms] [Info] [omni.server] [28868] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:29:36 [1ms] [Info] [omni.server] [28868] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:29:36 [1ms] [Info] [omni.server] [28868] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:29:36 [2ms] [Info] [omni.server] [28868] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:29:36 [2ms] [Info] [carb] [28868] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:29:36 [2ms] [Info] [carb] [28868] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:29:36 [2ms] [Info] [carb] [28868] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:29:36 [2ms] [Info] [carb] [28868] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:29:36 [2ms] [Info] [carb.launcher.plugin] [28868] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x24f0360 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:29:36 [2ms] [Info] [carb.launcher.plugin] [28868] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 28868. 2023-01-11 01:29:36 [2ms] [Info] [carb] [28868] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:29:36 [2ms] [Info] [carb] [28868] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:29:36 [3ms] [Info] [carb.launcher.plugin] [28880] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 28881 from the parent 28868. 2023-01-11 01:29:36 [3ms] [Info] [carb.launcher.plugin] [28880] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 28881. 2023-01-11 01:29:36 [4ms] [Info] [carb.launcher.plugin] [28880] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 28881. 2023-01-11 01:29:36 [4ms] [Info] [carb.launcher.plugin] [28868] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 28881 from the parent 28868. 2023-01-11 01:29:36 [4ms] [Info] [carb.launcher.plugin] [28868] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 28881 {process = 0x24f0360} 2023-01-11 01:29:36 [4ms] [Info] [omni.core] [28868] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:29:36 [4ms] [Info] [omni.core] [28868] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:29:36 [4ms] [Info] [carb.launcher.plugin] [28882] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 28881. 2023-01-11 01:29:36 [4ms] [Info] [carb.launcher.plugin] [28883] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 28881. 2023-01-11 01:29:36 [216ms] [Error] [omni.core] [28883] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:29:36 [219ms] [Error] [omni.core] [28883] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:29:36,442 - system_log - INFO - reconnecting ... 2023-01-11 01:29:36 [227ms] [Error] [omni.core] [28883] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:29:36,450 - system_log - INFO - connected 2023-01-11 01:29:36 [227ms] [Error] [omni.core] [28883] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [28881] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:29:36 [251ms] [Info] [carb.launcher.plugin] [28882] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:29:36 [251ms] [Info] [carb.launcher.plugin] [28882] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 28881, bytes = 0} 2023-01-11 01:29:36 [251ms] [Info] [carb.launcher.plugin] [28882] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 28881. 2023-01-11 01:29:36 [251ms] [Info] [carb.launcher.plugin] [28883] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:29:36 [251ms] [Info] [carb.launcher.plugin] [28883] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 28881, bytes = 0} 2023-01-11 01:29:36 [251ms] [Info] [carb.launcher.plugin] [28883] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 28881. 2023-01-11 01:29:37 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:29:37 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:29:37 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:29:37 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:29:37 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:29:37 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:29:37 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:29:37 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:29:37 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:29:37 [0ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:29:37 [1ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:29:37 [1ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:29:37 [1ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:29:37 [1ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:29:37 [1ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:29:37 [1ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:29:37 [1ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:29:37 [1ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:29:37 [1ms] [Info] [omni.server] [28889] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:29:37 [1ms] [Info] [omni.server] [28889] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:29:37 [2ms] [Info] [omni.server] [28889] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:29:37 [2ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:29:37 [2ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:29:37 [2ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:29:37 [2ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:29:37 [2ms] [Info] [carb.launcher.plugin] [28889] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x160ead0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:29:37 [2ms] [Info] [carb.launcher.plugin] [28889] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 28889. 2023-01-11 01:29:37 [2ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:29:37 [2ms] [Info] [carb] [28889] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:29:37 [3ms] [Info] [carb.launcher.plugin] [28901] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 28902 from the parent 28889. 2023-01-11 01:29:37 [3ms] [Info] [carb.launcher.plugin] [28901] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 28902. 2023-01-11 01:29:37 [3ms] [Info] [carb.launcher.plugin] [28901] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 28902. 2023-01-11 01:29:37 [3ms] [Info] [carb.launcher.plugin] [28889] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 28902 from the parent 28889. 2023-01-11 01:29:37 [3ms] [Info] [carb.launcher.plugin] [28889] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 28902 {process = 0x160ead0} 2023-01-11 01:29:37 [3ms] [Info] [omni.core] [28889] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:29:37 [3ms] [Info] [carb.launcher.plugin] [28904] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 28902. 2023-01-11 01:29:37 [3ms] [Info] [carb.launcher.plugin] [28903] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 28902. 2023-01-11 01:29:37 [3ms] [Info] [omni.core] [28889] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:29:37 [240ms] [Error] [omni.core] [28904] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:29:37 [243ms] [Error] [omni.core] [28904] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:29:37,492 - system_log - INFO - reconnecting ... 2023-01-11 01:29:37 [251ms] [Error] [omni.core] [28904] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:29:37,500 - system_log - INFO - connected 2023-01-11 01:29:37 [251ms] [Error] [omni.core] [28904] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [28902] 2023-01-11 01:29:37 [252ms] [Error] [omni.core] [28904] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:29:37 [276ms] [Info] [carb.launcher.plugin] [28904] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:29:37 [276ms] [Info] [carb.launcher.plugin] [28904] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 28902, bytes = 0} 2023-01-11 01:29:37 [276ms] [Info] [carb.launcher.plugin] [28904] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 28902. 2023-01-11 01:29:37 [276ms] [Info] [carb.launcher.plugin] [28903] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:29:37 [276ms] [Info] [carb.launcher.plugin] [28903] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 28902, bytes = 0} 2023-01-11 01:29:37 [276ms] [Info] [carb.launcher.plugin] [28903] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 28902. 2023-01-11 01:29:39 [2,004ms] [Info] [omni.core] [28895] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:29:39 [2,004ms] [Info] [omni.core] [28895] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:29:39 [2,004ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:29:39 [2,006ms] [Info] [omni.core] [28889] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:29:39 [2,007ms] [Info] [carb] [28889] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:29:39 [2,007ms] [Warning] [omni.keyvaluedatabase.leveldb] [28889] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:29:39 [2,008ms] [Info] [omni.core] [28895] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:29:39 [2,045ms] [Warning] [omni.keyvaluedatabase.leveldb] [28889] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:29:39 [2,046ms] [Fatal] [omni.core] [28889] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:29:45 [0ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:29:45 [1ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:29:45 [1ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:29:45 [1ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:29:45 [1ms] [Info] [carb] [28935] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:29:45 [1ms] [Info] [carb] [28935] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:29:45 [1ms] [Info] [carb] [28935] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:29:45 [1ms] [Info] [carb] [28935] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:29:45 [1ms] [Info] [carb] [28935] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:29:45 [1ms] [Info] [carb] [28935] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:29:45 [1ms] [Info] [omni.server] [28935] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:29:45 [1ms] [Info] [omni.server] [28935] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:29:45 [1ms] [Info] [omni.server] [28935] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:29:45 [1ms] [Info] [omni.server] [28935] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:29:45 [1ms] [Info] [omni.server] [28935] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:29:45 [1ms] [Info] [omni.server] [28935] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:29:45 [1ms] [Info] [omni.server] [28935] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:29:45 [1ms] [Info] [omni.server] [28935] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:29:45 [1ms] [Info] [omni.server] [28935] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:29:45 [1ms] [Info] [omni.server] [28935] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:29:45 [1ms] [Info] [carb] [28935] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:29:45 [2ms] [Info] [carb] [28935] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:29:45 [2ms] [Info] [carb] [28935] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:29:45 [2ms] [Info] [carb] [28935] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:29:45 [2ms] [Info] [carb.launcher.plugin] [28935] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x257aae0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:29:45 [2ms] [Info] [carb.launcher.plugin] [28935] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 28935. 2023-01-11 01:29:45 [2ms] [Info] [carb] [28935] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:29:45 [2ms] [Info] [carb] [28935] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:29:45 [3ms] [Info] [carb.launcher.plugin] [28947] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 28948 from the parent 28935. 2023-01-11 01:29:45 [3ms] [Info] [carb.launcher.plugin] [28947] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 28948. 2023-01-11 01:29:45 [3ms] [Info] [carb.launcher.plugin] [28947] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 28948. 2023-01-11 01:29:45 [3ms] [Info] [carb.launcher.plugin] [28935] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 28948 from the parent 28935. 2023-01-11 01:29:45 [3ms] [Info] [carb.launcher.plugin] [28935] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 28948 {process = 0x257aae0} 2023-01-11 01:29:45 [3ms] [Info] [omni.core] [28935] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:29:45 [3ms] [Info] [omni.core] [28935] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:29:45 [3ms] [Info] [carb.launcher.plugin] [28949] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 28948. 2023-01-11 01:29:45 [3ms] [Info] [carb.launcher.plugin] [28950] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 28948. 2023-01-11 01:29:45 [218ms] [Error] [omni.core] [28950] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:29:45 [221ms] [Error] [omni.core] [28950] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:29:45,759 - system_log - INFO - reconnecting ... 2023-01-11 01:29:45 [229ms] [Error] [omni.core] [28950] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:29:45,767 - system_log - INFO - connected 2023-01-11 01:29:45 [229ms] [Error] [omni.core] [28950] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [28948] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:29:45 [254ms] [Info] [carb.launcher.plugin] [28949] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:29:45 [254ms] [Info] [carb.launcher.plugin] [28950] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:29:45 [254ms] [Info] [carb.launcher.plugin] [28950] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 28948, bytes = 0} 2023-01-11 01:29:45 [254ms] [Info] [carb.launcher.plugin] [28949] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 28948, bytes = 0} 2023-01-11 01:29:45 [254ms] [Info] [carb.launcher.plugin] [28949] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 28948. 2023-01-11 01:29:45 [254ms] [Info] [carb.launcher.plugin] [28950] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 28948. 2023-01-11 01:29:46 [0ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:29:46 [0ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:29:46 [0ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:29:46 [0ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:29:46 [0ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:29:46 [0ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:29:46 [0ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:29:46 [0ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:29:46 [0ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:29:46 [0ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:29:46 [1ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:29:46 [1ms] [Info] [omni.server] [28956] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:29:46 [1ms] [Info] [omni.server] [28956] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:29:46 [1ms] [Info] [omni.server] [28956] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:29:46 [1ms] [Info] [omni.server] [28956] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:29:46 [1ms] [Info] [omni.server] [28956] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:29:46 [1ms] [Info] [omni.server] [28956] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:29:46 [1ms] [Info] [omni.server] [28956] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:29:46 [1ms] [Info] [omni.server] [28956] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:29:46 [1ms] [Info] [omni.server] [28956] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:29:46 [2ms] [Info] [omni.server] [28956] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:29:46 [2ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:29:46 [2ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:29:46 [2ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:29:46 [2ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:29:46 [2ms] [Info] [carb.launcher.plugin] [28956] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1612ac0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:29:46 [2ms] [Info] [carb.launcher.plugin] [28956] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 28956. 2023-01-11 01:29:46 [2ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:29:46 [2ms] [Info] [carb] [28956] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:29:46 [3ms] [Info] [carb.launcher.plugin] [28968] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 28969 from the parent 28956. 2023-01-11 01:29:46 [3ms] [Info] [carb.launcher.plugin] [28968] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 28969. 2023-01-11 01:29:46 [3ms] [Info] [carb.launcher.plugin] [28968] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 28969. 2023-01-11 01:29:46 [3ms] [Info] [carb.launcher.plugin] [28956] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 28969 from the parent 28956. 2023-01-11 01:29:46 [4ms] [Info] [carb.launcher.plugin] [28956] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 28969 {process = 0x1612ac0} 2023-01-11 01:29:46 [4ms] [Info] [omni.core] [28956] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:29:46 [4ms] [Info] [omni.core] [28956] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:29:46 [4ms] [Info] [carb.launcher.plugin] [28970] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 28969. 2023-01-11 01:29:46 [4ms] [Info] [carb.launcher.plugin] [28971] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 28969. 2023-01-11 01:29:46 [220ms] [Error] [omni.core] [28971] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:29:46 [224ms] [Error] [omni.core] [28971] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:29:46,381 - system_log - INFO - reconnecting ... 2023-01-11 01:29:46 [232ms] [Error] [omni.core] [28971] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:29:46,389 - system_log - INFO - connected 2023-01-11 01:29:46 [232ms] [Error] [omni.core] [28971] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [28969] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:29:46 [259ms] [Info] [carb.launcher.plugin] [28970] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:29:46 [259ms] [Info] [carb.launcher.plugin] [28971] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:29:46 [259ms] [Info] [carb.launcher.plugin] [28971] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 28969, bytes = 0} 2023-01-11 01:29:46 [259ms] [Info] [carb.launcher.plugin] [28970] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 28969, bytes = 0} 2023-01-11 01:29:46 [259ms] [Info] [carb.launcher.plugin] [28971] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 28969. 2023-01-11 01:29:46 [259ms] [Info] [carb.launcher.plugin] [28970] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 28969. 2023-01-11 01:29:48 [2,004ms] [Info] [omni.core] [28961] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:29:48 [2,004ms] [Info] [omni.core] [28961] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:29:48 [2,004ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:29:48 [2,006ms] [Info] [omni.core] [28956] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:29:48 [2,007ms] [Info] [carb] [28956] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:29:48 [2,007ms] [Warning] [omni.keyvaluedatabase.leveldb] [28956] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:29:48 [2,008ms] [Info] [omni.core] [28961] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:29:48 [2,042ms] [Warning] [omni.keyvaluedatabase.leveldb] [28956] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:29:48 [2,042ms] [Fatal] [omni.core] [28956] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-11 01:30:05 [0ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:30:05 [0ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:30:05 [0ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:05 [0ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:05 [0ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:05 [0ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:30:05 [0ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:05 [0ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:30:05 [0ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:05 [1ms] [Info] [carb] [28991] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:05 [1ms] [Info] [omni.server] [28991] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:30:05 [1ms] [Info] [omni.server] [28991] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:30:05 [1ms] [Info] [omni.server] [28991] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:30:05 [1ms] [Info] [omni.server] [28991] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:30:05 [1ms] [Info] [omni.server] [28991] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:30:05 [1ms] [Info] [omni.server] [28991] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:30:05 [1ms] [Info] [omni.server] [28991] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:30:05 [1ms] [Info] [omni.server] [28991] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:30:05 [1ms] [Info] [omni.server] [28991] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:30:05 [2ms] [Info] [omni.server] [28991] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:30:05 [2ms] [Info] [carb] [28991] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:05 [2ms] [Info] [carb] [28991] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:05 [2ms] [Info] [carb] [28991] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:05 [2ms] [Info] [carb] [28991] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:05 [2ms] [Info] [carb.launcher.plugin] [28991] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xc8eae0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:30:05 [2ms] [Info] [carb.launcher.plugin] [28991] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 28991. 2023-01-11 01:30:05 [2ms] [Info] [carb] [28991] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:30:05 [2ms] [Info] [carb] [28991] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:30:05 [3ms] [Info] [carb.launcher.plugin] [29003] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 29004 from the parent 28991. 2023-01-11 01:30:05 [3ms] [Info] [carb.launcher.plugin] [29003] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 29004. 2023-01-11 01:30:05 [4ms] [Info] [carb.launcher.plugin] [29003] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 29004. 2023-01-11 01:30:05 [4ms] [Info] [carb.launcher.plugin] [28991] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 29004 from the parent 28991. 2023-01-11 01:30:05 [4ms] [Info] [carb.launcher.plugin] [28991] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 29004 {process = 0xc8eae0} 2023-01-11 01:30:05 [4ms] [Info] [carb.launcher.plugin] [29005] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 29004. 2023-01-11 01:30:05 [4ms] [Info] [omni.core] [28991] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:30:05 [4ms] [Info] [omni.core] [28991] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:30:05 [4ms] [Info] [carb.launcher.plugin] [29006] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 29004. 2023-01-11 01:30:06 [214ms] [Error] [omni.core] [29006] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:30:06 [217ms] [Error] [omni.core] [29006] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:06,168 - system_log - INFO - reconnecting ... 2023-01-11 01:30:06 [225ms] [Error] [omni.core] [29006] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:06,176 - system_log - INFO - connected 2023-01-11 01:30:06 [225ms] [Error] [omni.core] [29006] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [29004] 2023-01-11 01:30:06 [225ms] [Error] [omni.core] [29006] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:30:06 [249ms] [Info] [carb.launcher.plugin] [29006] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:06 [249ms] [Info] [carb.launcher.plugin] [29006] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29004, bytes = 0} 2023-01-11 01:30:06 [249ms] [Info] [carb.launcher.plugin] [29006] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 29004. 2023-01-11 01:30:06 [249ms] [Info] [carb.launcher.plugin] [29005] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:06 [249ms] [Info] [carb.launcher.plugin] [29005] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29004, bytes = 0} 2023-01-11 01:30:06 [249ms] [Info] [carb.launcher.plugin] [29005] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 29004. 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:30:07 [0ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:07 [1ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:30:07 [1ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:07 [1ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:30:07 [1ms] [Info] [carb] [29014] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:07 [1ms] [Info] [carb] [29014] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:07 [1ms] [Info] [carb] [29014] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:07 [1ms] [Info] [carb] [29014] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:07 [1ms] [Info] [carb] [29014] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:07 [1ms] [Info] [carb] [29014] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:07 [1ms] [Info] [omni.server] [29014] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:30:07 [1ms] [Info] [omni.server] [29014] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:30:07 [1ms] [Info] [omni.server] [29014] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:30:07 [1ms] [Info] [omni.server] [29014] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:30:07 [1ms] [Info] [omni.server] [29014] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:30:07 [1ms] [Info] [omni.server] [29014] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:30:07 [1ms] [Info] [omni.server] [29014] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:30:07 [1ms] [Info] [omni.server] [29014] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:30:07 [1ms] [Info] [omni.server] [29014] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:30:07 [1ms] [Info] [omni.server] [29014] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:30:07 [1ms] [Info] [carb] [29014] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:07 [2ms] [Info] [carb] [29014] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:07 [2ms] [Info] [carb] [29014] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:07 [2ms] [Info] [carb] [29014] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:07 [2ms] [Info] [carb.launcher.plugin] [29014] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x2291ad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:30:07 [2ms] [Info] [carb.launcher.plugin] [29014] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 29014. 2023-01-11 01:30:07 [2ms] [Info] [carb] [29014] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:30:07 [2ms] [Info] [carb] [29014] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:30:07 [3ms] [Info] [carb.launcher.plugin] [29026] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 29027 from the parent 29014. 2023-01-11 01:30:07 [3ms] [Info] [carb.launcher.plugin] [29026] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 29027. 2023-01-11 01:30:07 [3ms] [Info] [carb.launcher.plugin] [29026] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 29027. 2023-01-11 01:30:07 [3ms] [Info] [carb.launcher.plugin] [29014] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 29027 from the parent 29014. 2023-01-11 01:30:07 [3ms] [Info] [carb.launcher.plugin] [29014] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 29027 {process = 0x2291ad0} 2023-01-11 01:30:07 [3ms] [Info] [omni.core] [29014] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:30:07 [3ms] [Info] [omni.core] [29014] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:30:07 [3ms] [Info] [carb.launcher.plugin] [29029] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 29027. 2023-01-11 01:30:07 [3ms] [Info] [carb.launcher.plugin] [29028] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 29027. 2023-01-11 01:30:07 [220ms] [Error] [omni.core] [29029] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:30:07 [224ms] [Error] [omni.core] [29029] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:07,243 - system_log - INFO - reconnecting ... 2023-01-11 01:30:07 [232ms] [Error] [omni.core] [29029] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:07,251 - system_log - INFO - connected 2023-01-11 01:30:07 [232ms] [Error] [omni.core] [29029] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [29027] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:30:07 [257ms] [Info] [carb.launcher.plugin] [29028] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:07 [257ms] [Info] [carb.launcher.plugin] [29029] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:07 [257ms] [Info] [carb.launcher.plugin] [29029] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29027, bytes = 0} 2023-01-11 01:30:07 [257ms] [Info] [carb.launcher.plugin] [29028] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29027, bytes = 0} 2023-01-11 01:30:07 [257ms] [Info] [carb.launcher.plugin] [29029] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 29027. 2023-01-11 01:30:07 [257ms] [Info] [carb.launcher.plugin] [29028] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 29027. 2023-01-11 01:30:08 [0ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:30:08 [0ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:30:08 [0ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:08 [0ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:08 [0ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:08 [0ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:30:08 [0ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:08 [0ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:30:08 [0ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:08 [0ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:08 [1ms] [Info] [carb] [29036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:08 [1ms] [Info] [omni.server] [29036] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:30:08 [1ms] [Info] [omni.server] [29036] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:30:08 [1ms] [Info] [omni.server] [29036] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:30:08 [1ms] [Info] [omni.server] [29036] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:30:08 [1ms] [Info] [omni.server] [29036] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:30:08 [1ms] [Info] [omni.server] [29036] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:30:08 [1ms] [Info] [omni.server] [29036] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:30:08 [1ms] [Info] [omni.server] [29036] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:30:08 [1ms] [Info] [omni.server] [29036] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:30:08 [2ms] [Info] [omni.server] [29036] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:30:08 [2ms] [Info] [carb] [29036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:08 [2ms] [Info] [carb] [29036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:08 [2ms] [Info] [carb] [29036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:08 [2ms] [Info] [carb] [29036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:08 [2ms] [Info] [carb.launcher.plugin] [29036] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x22e4ae0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:30:08 [2ms] [Info] [carb.launcher.plugin] [29036] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 29036. 2023-01-11 01:30:08 [2ms] [Info] [carb] [29036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:30:08 [2ms] [Info] [carb] [29036] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:30:08 [3ms] [Info] [carb.launcher.plugin] [29048] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 29049 from the parent 29036. 2023-01-11 01:30:08 [3ms] [Info] [carb.launcher.plugin] [29048] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 29049. 2023-01-11 01:30:08 [3ms] [Info] [carb.launcher.plugin] [29048] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 29049. 2023-01-11 01:30:08 [3ms] [Info] [carb.launcher.plugin] [29036] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 29049 from the parent 29036. 2023-01-11 01:30:08 [3ms] [Info] [carb.launcher.plugin] [29036] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 29049 {process = 0x22e4ae0} 2023-01-11 01:30:08 [3ms] [Info] [omni.core] [29036] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:30:08 [3ms] [Info] [omni.core] [29036] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:30:08 [3ms] [Info] [carb.launcher.plugin] [29050] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 29049. 2023-01-11 01:30:08 [3ms] [Info] [carb.launcher.plugin] [29051] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 29049. 2023-01-11 01:30:08 [214ms] [Error] [omni.core] [29051] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:30:08 [217ms] [Error] [omni.core] [29051] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:08,937 - system_log - INFO - reconnecting ... 2023-01-11 01:30:08 [225ms] [Error] [omni.core] [29051] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:08,945 - system_log - INFO - connected 2023-01-11 01:30:08 [226ms] [Error] [omni.core] [29051] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [29049] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:30:08 [250ms] [Info] [carb.launcher.plugin] [29050] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:08 [250ms] [Info] [carb.launcher.plugin] [29050] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29049, bytes = 0} 2023-01-11 01:30:08 [250ms] [Info] [carb.launcher.plugin] [29050] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 29049. 2023-01-11 01:30:08 [250ms] [Info] [carb.launcher.plugin] [29051] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:08 [250ms] [Info] [carb.launcher.plugin] [29051] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29049, bytes = 0} 2023-01-11 01:30:08 [250ms] [Info] [carb.launcher.plugin] [29051] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 29049. 2023-01-11 01:30:10 [0ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:30:10 [0ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:30:10 [0ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:10 [0ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:10 [0ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:10 [0ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:30:10 [0ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:10 [0ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:30:10 [0ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:10 [0ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:10 [1ms] [Info] [carb] [29058] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:10 [1ms] [Info] [omni.server] [29058] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:30:10 [1ms] [Info] [omni.server] [29058] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:30:10 [1ms] [Info] [omni.server] [29058] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:30:10 [1ms] [Info] [omni.server] [29058] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:30:10 [1ms] [Info] [omni.server] [29058] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:30:10 [1ms] [Info] [omni.server] [29058] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:30:10 [1ms] [Info] [omni.server] [29058] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:30:10 [1ms] [Info] [omni.server] [29058] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:30:10 [1ms] [Info] [omni.server] [29058] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:30:10 [2ms] [Info] [omni.server] [29058] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:30:10 [2ms] [Info] [carb] [29058] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:10 [2ms] [Info] [carb] [29058] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:10 [2ms] [Info] [carb] [29058] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:10 [2ms] [Info] [carb] [29058] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:10 [2ms] [Info] [carb.launcher.plugin] [29058] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x2213ad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:30:10 [2ms] [Info] [carb.launcher.plugin] [29058] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 29058. 2023-01-11 01:30:10 [2ms] [Info] [carb] [29058] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:30:10 [2ms] [Info] [carb] [29058] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:30:10 [3ms] [Info] [carb.launcher.plugin] [29070] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 29071 from the parent 29058. 2023-01-11 01:30:10 [3ms] [Info] [carb.launcher.plugin] [29070] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 29071. 2023-01-11 01:30:10 [3ms] [Info] [carb.launcher.plugin] [29070] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 29071. 2023-01-11 01:30:10 [3ms] [Info] [carb.launcher.plugin] [29058] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 29071 from the parent 29058. 2023-01-11 01:30:10 [3ms] [Info] [carb.launcher.plugin] [29058] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 29071 {process = 0x2213ad0} 2023-01-11 01:30:10 [3ms] [Info] [omni.core] [29058] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:30:10 [3ms] [Info] [carb.launcher.plugin] [29072] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 29071. 2023-01-11 01:30:10 [3ms] [Info] [omni.core] [29058] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:30:10 [3ms] [Info] [carb.launcher.plugin] [29073] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 29071. 2023-01-11 01:30:10 [215ms] [Error] [omni.core] [29073] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:30:10 [218ms] [Error] [omni.core] [29073] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:10,284 - system_log - INFO - reconnecting ... 2023-01-11 01:30:10 [225ms] [Error] [omni.core] [29073] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:10,292 - system_log - INFO - connected 2023-01-11 01:30:10 [226ms] [Error] [omni.core] [29073] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [29071] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:30:10 [251ms] [Info] [carb.launcher.plugin] [29072] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:10 [251ms] [Info] [carb.launcher.plugin] [29072] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29071, bytes = 0} 2023-01-11 01:30:10 [251ms] [Info] [carb.launcher.plugin] [29072] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 29071. 2023-01-11 01:30:10 [251ms] [Info] [carb.launcher.plugin] [29073] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:10 [251ms] [Info] [carb.launcher.plugin] [29073] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29071, bytes = 0} 2023-01-11 01:30:10 [251ms] [Info] [carb.launcher.plugin] [29073] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 29071. 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:11 [0ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:30:11 [1ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:11 [1ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:30:11 [1ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:11 [1ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:30:11 [1ms] [Info] [carb] [29081] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:11 [1ms] [Info] [carb] [29081] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:11 [1ms] [Info] [carb] [29081] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:11 [1ms] [Info] [carb] [29081] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:11 [1ms] [Info] [carb] [29081] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:11 [1ms] [Info] [carb] [29081] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:11 [1ms] [Info] [omni.server] [29081] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:30:11 [1ms] [Info] [omni.server] [29081] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:30:11 [1ms] [Info] [omni.server] [29081] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:30:11 [1ms] [Info] [omni.server] [29081] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:30:11 [1ms] [Info] [omni.server] [29081] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:30:11 [1ms] [Info] [omni.server] [29081] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:30:11 [1ms] [Info] [omni.server] [29081] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:30:11 [1ms] [Info] [omni.server] [29081] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:30:11 [1ms] [Info] [omni.server] [29081] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:30:11 [1ms] [Info] [omni.server] [29081] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:30:11 [2ms] [Info] [carb] [29081] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:11 [2ms] [Info] [carb] [29081] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:11 [2ms] [Info] [carb] [29081] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:11 [2ms] [Info] [carb] [29081] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:11 [2ms] [Info] [carb.launcher.plugin] [29081] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x20a9ad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:30:11 [2ms] [Info] [carb.launcher.plugin] [29081] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 29081. 2023-01-11 01:30:11 [2ms] [Info] [carb] [29081] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:30:11 [2ms] [Info] [carb] [29081] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:30:11 [3ms] [Info] [carb.launcher.plugin] [29093] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 29094 from the parent 29081. 2023-01-11 01:30:11 [3ms] [Info] [carb.launcher.plugin] [29093] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 29094. 2023-01-11 01:30:11 [3ms] [Info] [carb.launcher.plugin] [29093] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 29094. 2023-01-11 01:30:11 [3ms] [Info] [carb.launcher.plugin] [29081] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 29094 from the parent 29081. 2023-01-11 01:30:11 [3ms] [Info] [carb.launcher.plugin] [29081] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 29094 {process = 0x20a9ad0} 2023-01-11 01:30:11 [3ms] [Info] [omni.core] [29081] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:30:11 [3ms] [Info] [omni.core] [29081] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:30:11 [3ms] [Info] [carb.launcher.plugin] [29095] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 29094. 2023-01-11 01:30:11 [3ms] [Info] [carb.launcher.plugin] [29096] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 29094. 2023-01-11 01:30:11 [214ms] [Error] [omni.core] [29096] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:30:11 [217ms] [Error] [omni.core] [29096] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:11,447 - system_log - INFO - reconnecting ... 2023-01-11 01:30:11 [225ms] [Error] [omni.core] [29096] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:11,455 - system_log - INFO - connected 2023-01-11 01:30:11 [225ms] [Error] [omni.core] [29096] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [29094] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:30:11 [250ms] [Info] [carb.launcher.plugin] [29095] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:11 [250ms] [Info] [carb.launcher.plugin] [29096] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:11 [250ms] [Info] [carb.launcher.plugin] [29096] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29094, bytes = 0} 2023-01-11 01:30:11 [250ms] [Info] [carb.launcher.plugin] [29095] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29094, bytes = 0} 2023-01-11 01:30:11 [250ms] [Info] [carb.launcher.plugin] [29095] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 29094. 2023-01-11 01:30:11 [250ms] [Info] [carb.launcher.plugin] [29096] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 29094. 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:12 [0ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:30:12 [1ms] [Info] [omni.server] [29103] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:30:12 [1ms] [Info] [omni.server] [29103] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:30:12 [1ms] [Info] [omni.server] [29103] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:30:12 [1ms] [Info] [omni.server] [29103] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:30:12 [1ms] [Info] [omni.server] [29103] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:30:12 [1ms] [Info] [omni.server] [29103] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:30:12 [1ms] [Info] [omni.server] [29103] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:30:12 [1ms] [Info] [omni.server] [29103] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:30:12 [1ms] [Info] [omni.server] [29103] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:30:12 [1ms] [Info] [omni.server] [29103] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:30:12 [1ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:30:12 [2ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:30:12 [2ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:30:12 [2ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:30:12 [2ms] [Info] [carb.launcher.plugin] [29103] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xa7bae0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:30:12 [2ms] [Info] [carb.launcher.plugin] [29103] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 29103. 2023-01-11 01:30:12 [2ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:30:12 [2ms] [Info] [carb] [29103] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:30:12 [3ms] [Info] [carb.launcher.plugin] [29115] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 29116 from the parent 29103. 2023-01-11 01:30:12 [3ms] [Info] [carb.launcher.plugin] [29115] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 29116. 2023-01-11 01:30:12 [3ms] [Info] [carb.launcher.plugin] [29115] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 29116. 2023-01-11 01:30:12 [3ms] [Info] [carb.launcher.plugin] [29103] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 29116 from the parent 29103. 2023-01-11 01:30:12 [3ms] [Info] [carb.launcher.plugin] [29103] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 29116 {process = 0xa7bae0} 2023-01-11 01:30:12 [3ms] [Info] [omni.core] [29103] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:30:12 [3ms] [Info] [omni.core] [29103] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:30:12 [3ms] [Info] [carb.launcher.plugin] [29117] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 29116. 2023-01-11 01:30:12 [3ms] [Info] [carb.launcher.plugin] [29118] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 29116. 2023-01-11 01:30:12 [220ms] [Error] [omni.core] [29118] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:30:12 [223ms] [Error] [omni.core] [29118] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:12,628 - system_log - INFO - reconnecting ... 2023-01-11 01:30:12 [230ms] [Error] [omni.core] [29118] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:30:12,636 - system_log - INFO - connected 2023-01-11 01:30:12 [231ms] [Error] [omni.core] [29118] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [29116] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:30:12 [257ms] [Info] [carb.launcher.plugin] [29117] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:12 [257ms] [Info] [carb.launcher.plugin] [29117] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29116, bytes = 0} 2023-01-11 01:30:12 [257ms] [Info] [carb.launcher.plugin] [29117] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 29116. 2023-01-11 01:30:12 [257ms] [Info] [carb.launcher.plugin] [29118] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:30:12 [257ms] [Info] [carb.launcher.plugin] [29118] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 29116, bytes = 0} 2023-01-11 01:30:12 [257ms] [Info] [carb.launcher.plugin] [29118] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 29116. 2023-01-11 01:30:14 [2,004ms] [Info] [omni.core] [29108] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:30:14 [2,004ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:30:14 [2,004ms] [Info] [omni.core] [29108] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:30:14 [2,006ms] [Info] [omni.core] [29103] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:30:14 [2,006ms] [Info] [carb] [29103] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:30:14 [2,007ms] [Warning] [omni.keyvaluedatabase.leveldb] [29103] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:30:14 [2,008ms] [Info] [omni.core] [29108] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:30:14 [2,035ms] [Warning] [omni.keyvaluedatabase.leveldb] [29103] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:30:14 [2,035ms] [Fatal] [omni.core] [29103] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:36:10 [0ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:36:10 [1ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:36:10 [1ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:36:10 [1ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:36:10 [1ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:36:10 [1ms] [Info] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:36:10 [1ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:36:10 [1ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:36:10 [1ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:36:10 [1ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:36:10 [1ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:36:10 [1ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:36:10 [1ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:36:10 [1ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:36:10 [1ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:36:10 [1ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:36:10 [1ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:36:10 [1ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:36:10 [1ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:36:10 [1ms] [Info] [omni.server] [31794] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:36:10 [2ms] [Info] [omni.server] [31794] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:36:10 [2ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:36:10 [2ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:36:10 [2ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:36:10 [2ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:36:10 [2ms] [Info] [carb.launcher.plugin] [31794] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1e5abb0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:36:10 [2ms] [Info] [carb.launcher.plugin] [31794] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 31794. 2023-01-11 01:36:10 [2ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:36:10 [2ms] [Info] [carb] [31794] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:36:10 [3ms] [Info] [carb.launcher.plugin] [31810] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 31811 from the parent 31794. 2023-01-11 01:36:10 [3ms] [Info] [carb.launcher.plugin] [31810] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 31811. 2023-01-11 01:36:10 [3ms] [Info] [carb.launcher.plugin] [31810] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 31811. 2023-01-11 01:36:10 [3ms] [Info] [carb.launcher.plugin] [31794] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 31811 from the parent 31794. 2023-01-11 01:36:10 [3ms] [Info] [carb.launcher.plugin] [31794] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 31811 {process = 0x1e5abb0} 2023-01-11 01:36:10 [3ms] [Info] [omni.core] [31794] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:36:10 [3ms] [Info] [omni.core] [31794] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:36:10 [3ms] [Info] [carb.launcher.plugin] [31812] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 31811. 2023-01-11 01:36:10 [3ms] [Info] [carb.launcher.plugin] [31813] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 31811. 2023-01-11 01:36:11 [313ms] [Error] [omni.core] [31813] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:36:11 [318ms] [Error] [omni.core] [31813] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:36:11,309 - system_log - INFO - reconnecting ... 2023-01-11 01:36:11 [328ms] [Error] [omni.core] [31813] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:36:11,319 - system_log - INFO - connected 2023-01-11 01:36:11 [328ms] [Error] [omni.core] [31813] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [31811] 2023-01-11 01:36:11 [328ms] [Error] [omni.core] [31813] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:36:11 [363ms] [Info] [carb.launcher.plugin] [31813] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:36:11 [363ms] [Info] [carb.launcher.plugin] [31812] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:36:11 [363ms] [Info] [carb.launcher.plugin] [31813] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 31811, bytes = 0} 2023-01-11 01:36:11 [363ms] [Info] [carb.launcher.plugin] [31812] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 31811, bytes = 0} 2023-01-11 01:36:11 [363ms] [Info] [carb.launcher.plugin] [31813] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 31811. 2023-01-11 01:36:11 [363ms] [Info] [carb.launcher.plugin] [31812] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 31811. 2023-01-11 01:36:11 [69,851,254ms] [Info] [omni.server.app] [3377] ../../../source/apps/omni.server/main.cpp:main():106: Got signal to exit the application. 2023-01-11 01:36:11 [69,851,254ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():391: OmniverseServer shutdown process starting. 2023-01-11 01:36:11 [69,851,254ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():392: Waiting for any outstanding transactions to complete. 2023-01-11 01:36:11 [69,851,254ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:beginShutdown():396: Releasing WebSocket listen ports 2023-01-11 01:36:11 [69,851,254ms] [Info] [omni.server] [31930] ../../../source/plugins/omni.server/ApiConnection.cpp:shutdownCore():70: Shutdown Core 2023-01-11 01:36:11 [69,851,254ms] [Info] [omni.core] [3899] ../../../source/plugins/omni.core/impl/Scheduler.cpp:scheduler():122: Scheduling task is shutted down... 2023-01-11 01:36:11 [69,851,254ms] [Info] [omni.core] [3386] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():422: Finished content remove task 2023-01-11 01:36:11 [69,851,254ms] [Error] [omni.core] [3394] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Shutting down server...wait 2023-01-11 01:36:11 [69,851,254ms] [Warning] [carb] [31930] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-11 01:36:11 [69,851,256ms] [Info] [omni.core] [31930] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:operator()():849: Content resolver statistics: Total: flush: 23 write: 3 read: 8 stat: 0 remove: 1 list: 0|Temporal: flush: 0 write: 3 read: 2 stat: 0 remove: 3 list: 0|Persistent: flush: 23 write: 2 read: 8 stat: 0 remove: 0 list: 0 2023-01-11 01:36:11 [69,851,256ms] [Info] [omni.server.app] [3377] ../../../source/apps/omni.server/main.cpp:main():120: Releasing the OmniverseServer instance. 2023-01-11 01:36:11 [69,851,256ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():310: OmniverseServerImpl destructor. 2023-01-11 01:36:11 [69,851,256ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():312: Releasing ClientConnectionManager 2023-01-11 01:36:11 [69,851,256ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():315: Releasing ServiceConnectionManager 2023-01-11 01:36:11 [69,851,256ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():318: Releasing LftConnectionManager 2023-01-11 01:36:11 [69,851,256ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():321: Releasing ConnectionManager 2023-01-11 01:36:11 [69,851,256ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 3940 {exitCode = 137, flags = 0x00000002, timeout = 5000} 2023-01-11 01:36:11 [69,851,257ms] [Info] [carb.launcher.plugin] [3941] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:36:11 [69,851,257ms] [Info] [carb.launcher.plugin] [3942] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:36:11 [69,851,257ms] [Info] [carb.launcher.plugin] [3941] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3940, bytes = 0} 2023-01-11 01:36:11 [69,851,257ms] [Info] [carb.launcher.plugin] [3942] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3940, bytes = 0} 2023-01-11 01:36:11 [69,851,257ms] [Info] [carb.launcher.plugin] [3941] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 3940. 2023-01-11 01:36:11 [69,851,257ms] [Info] [carb.launcher.plugin] [3942] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 3940. 2023-01-11 01:36:11 [69,851,259ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x7f2af0003e00 (3940) to exit {timeout = 18446744073709551615} 2023-01-11 01:36:11 [69,851,259ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-01-11 01:36:11 [69,851,259ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x7f2af0003e00 {pid = 3940} 2023-01-11 01:36:11 [69,851,259ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x7f2af0003e00 {pid = 3940} 2023-01-11 01:36:11 [69,851,259ms] [Info] [omni.core] [3384] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():457: discovery client: exiting 2023-01-11 01:36:11 [69,851,259ms] [Info] [omni.core] [3377] ../../../source/plugins/common/ChildProcess.cpp:kill():125: Killing child server process ./omni-lft-service/omni-lft-service 2023-01-11 01:36:11 [69,851,259ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 3392 {exitCode = 137, flags = 0x00000003, timeout = 5000} 2023-01-11 01:36:11 [69,851,260ms] [Info] [carb.launcher.plugin] [3394] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:36:11 [69,851,260ms] [Info] [carb.launcher.plugin] [3394] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3392, bytes = 0} 2023-01-11 01:36:11 [69,851,260ms] [Info] [carb.launcher.plugin] [3394] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 3392. 2023-01-11 01:36:11 [69,851,260ms] [Info] [carb.launcher.plugin] [3393] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:36:11 [69,851,260ms] [Info] [carb.launcher.plugin] [3393] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 3392, bytes = 0} 2023-01-11 01:36:11 [69,851,260ms] [Info] [carb.launcher.plugin] [3393] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 3392. 2023-01-11 01:36:11 [69,851,262ms] [Info] [omni.core] [3377] ../../../source/plugins/common/ChildProcess.cpp:kill():127: Waiting for child server process ./omni-lft-service/omni-lft-service to exit 2023-01-11 01:36:11 [69,851,262ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0xb8e0c0 (3392) to exit {timeout = 18446744073709551615} 2023-01-11 01:36:11 [69,851,262ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-01-11 01:36:11 [69,851,262ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0xb8e0c0 {pid = 3392} 2023-01-11 01:36:11 [69,851,262ms] [Info] [carb.launcher.plugin] [3377] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0xb8e0c0 {pid = 3392} 2023-01-11 01:36:11 [69,851,262ms] [Info] [omni.server] [3377] ../../../source/plugins/omni.server/OmniverseServer.cpp:destroy():327: Destructor complete. 2023-01-11 01:36:12 [2,004ms] [Info] [omni.core] [31803] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:36:12 [2,004ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:36:12 [2,004ms] [Info] [omni.core] [31803] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:36:12 [2,006ms] [Info] [omni.core] [31794] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:36:12 [2,006ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:36:12 [2,008ms] [Info] [omni.core] [31803] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:36:13 [2,088ms] [Warning] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IObjects v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-11 01:36:13 [2,088ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:36:13 [2,138ms] [Warning] [carb] [31794] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():747: Acquiring non optional plugin interface which is not listed as dependency: [omni::server::IMetaUpgrade v2.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client. 2023-01-11 01:36:13 [2,138ms] [Info] [carb] [31794] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:36:13 [2,138ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2915: Meta database doesn't require upgrade 2023-01-11 01:36:13 [2,150ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1463: Fixing folder children counters .... 2023-01-11 01:36:13 [2,150ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1 / 17549 records 2023-01-11 01:36:13 [2,157ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 1001 / 17549 records 2023-01-11 01:36:13 [2,160ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 2001 / 17549 records 2023-01-11 01:36:13 [2,163ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 3001 / 17549 records 2023-01-11 01:36:13 [2,165ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 4001 / 17549 records 2023-01-11 01:36:13 [2,167ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 5001 / 17549 records 2023-01-11 01:36:13 [2,169ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 6001 / 17549 records 2023-01-11 01:36:13 [2,171ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 7001 / 17549 records 2023-01-11 01:36:13 [2,171ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 8001 / 17549 records 2023-01-11 01:36:13 [2,172ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 9001 / 17549 records 2023-01-11 01:36:13 [2,172ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 10001 / 17549 records 2023-01-11 01:36:13 [2,173ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 11001 / 17549 records 2023-01-11 01:36:13 [2,173ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 12001 / 17549 records 2023-01-11 01:36:13 [2,173ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 13001 / 17549 records 2023-01-11 01:36:13 [2,174ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 14001 / 17549 records 2023-01-11 01:36:13 [2,174ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 15001 / 17549 records 2023-01-11 01:36:13 [2,175ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 16001 / 17549 records 2023-01-11 01:36:13 [2,176ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1471: Processed 17001 / 17549 records 2023-01-11 01:36:13 [2,177ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1497: Checking for orphaned folder children counter records .... 2023-01-11 01:36:13 [2,178ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1 / 4247 records 2023-01-11 01:36:13 [2,179ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 1001 / 4247 records 2023-01-11 01:36:13 [2,180ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 2001 / 4247 records 2023-01-11 01:36:13 [2,181ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 3001 / 4247 records 2023-01-11 01:36:13 [2,183ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1508: Processed 4001 / 4247 records 2023-01-11 01:36:13 [2,183ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixFolderChildrenCounters():1532: Fixing folder children counters done 2023-01-11 01:36:13 [2,188ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1304: Fixing content refcounts .... 2023-01-11 01:36:13 [2,188ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1311: Collecting content -> refcount mapping ... 2023-01-11 01:36:13 [2,188ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1 / 17549 records 2023-01-11 01:36:13 [2,188ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 1001 / 17549 records 2023-01-11 01:36:13 [2,191ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 2001 / 17549 records 2023-01-11 01:36:13 [2,194ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 3001 / 17549 records 2023-01-11 01:36:13 [2,198ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 4001 / 17549 records 2023-01-11 01:36:13 [2,201ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 5001 / 17549 records 2023-01-11 01:36:13 [2,204ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 6001 / 17549 records 2023-01-11 01:36:13 [2,210ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 7001 / 17549 records 2023-01-11 01:36:13 [2,216ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 8001 / 17549 records 2023-01-11 01:36:13 [2,222ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 9001 / 17549 records 2023-01-11 01:36:13 [2,228ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 10001 / 17549 records 2023-01-11 01:36:13 [2,234ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 11001 / 17549 records 2023-01-11 01:36:13 [2,241ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 12001 / 17549 records 2023-01-11 01:36:13 [2,247ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 13001 / 17549 records 2023-01-11 01:36:13 [2,254ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 14001 / 17549 records 2023-01-11 01:36:13 [2,260ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 15001 / 17549 records 2023-01-11 01:36:13 [2,266ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 16001 / 17549 records 2023-01-11 01:36:13 [2,271ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1315: Processed 17001 / 17549 records 2023-01-11 01:36:13 [2,274ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1350: Collecting content -> refcount mapping done 2023-01-11 01:36:13 [2,274ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1351: Setting refcount ... 2023-01-11 01:36:13 [2,274ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1 / 7700 records 2023-01-11 01:36:13 [2,275ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 1001 / 7700 records 2023-01-11 01:36:13 [2,276ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 2001 / 7700 records 2023-01-11 01:36:13 [2,277ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 3001 / 7700 records 2023-01-11 01:36:13 [2,278ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 4001 / 7700 records 2023-01-11 01:36:13 [2,279ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 5001 / 7700 records 2023-01-11 01:36:13 [2,280ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 6001 / 7700 records 2023-01-11 01:36:13 [2,280ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1356: Processed 7001 / 7700 records 2023-01-11 01:36:13 [2,281ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1380: Setting refcount done 2023-01-11 01:36:13 [2,281ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixContentRefcounts():1382: Checking content table for unused content 2023-01-11 01:36:13 [2,288ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1550: Repairing versioning data... 2023-01-11 01:36:13 [2,288ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1563: Reading meta records... 2023-01-11 01:36:13 [2,289ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17549 records 2023-01-11 01:36:13 [2,289ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17549 records 2023-01-11 01:36:13 [2,289ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17549 records 2023-01-11 01:36:13 [2,289ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17549 records 2023-01-11 01:36:13 [2,289ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17549 records 2023-01-11 01:36:13 [2,289ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17549 records 2023-01-11 01:36:13 [2,289ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17549 records 2023-01-11 01:36:13 [2,289ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17549 records 2023-01-11 01:36:13 [2,290ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17549 records 2023-01-11 01:36:13 [2,290ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17549 records 2023-01-11 01:36:13 [2,291ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17549 records 2023-01-11 01:36:13 [2,291ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17549 records 2023-01-11 01:36:13 [2,291ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17549 records 2023-01-11 01:36:13 [2,291ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17549 records 2023-01-11 01:36:13 [2,292ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17549 records 2023-01-11 01:36:13 [2,292ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17549 records 2023-01-11 01:36:13 [2,293ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17549 records 2023-01-11 01:36:13 [2,299ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 1000/17549 records 2023-01-11 01:36:13 [2,305ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 2000/17549 records 2023-01-11 01:36:13 [2,311ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 3000/17549 records 2023-01-11 01:36:13 [2,318ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 4000/17549 records 2023-01-11 01:36:13 [2,324ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 5000/17549 records 2023-01-11 01:36:13 [2,330ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 6000/17549 records 2023-01-11 01:36:13 [2,338ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 7000/17549 records 2023-01-11 01:36:13 [2,347ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 8000/17549 records 2023-01-11 01:36:13 [2,355ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 9000/17549 records 2023-01-11 01:36:13 [2,363ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 10000/17549 records 2023-01-11 01:36:13 [2,370ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 11000/17549 records 2023-01-11 01:36:13 [2,378ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 12000/17549 records 2023-01-11 01:36:13 [2,387ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 13000/17549 records 2023-01-11 01:36:13 [2,395ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 14000/17549 records 2023-01-11 01:36:13 [2,404ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 15000/17549 records 2023-01-11 01:36:13 [2,413ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 16000/17549 records 2023-01-11 01:36:13 [2,419ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1575: Processed 17000/17549 records 2023-01-11 01:36:13 [2,422ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1671: Checking branch lists... 2023-01-11 01:36:13 [2,449ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1716: Fixed branch lists count: 0 2023-01-11 01:36:13 [2,449ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1718: Check for orphaned branch lists... 2023-01-11 01:36:13 [2,455ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1741: Checking version reference counters... 2023-01-11 01:36:13 [2,456ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1796: Fixed version reference counters: 0 2023-01-11 01:36:13 [2,456ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1797: Deleted orphaned version reference counters 0 2023-01-11 01:36:13 [2,456ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1799: Checking checkpoint messages... 2023-01-11 01:36:13 [2,456ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():1860: Verifying checkpoints... 2023-01-11 01:36:13 [2,456ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2090: Deleted checkpoints without HEAD: 0 2023-01-11 01:36:13 [2,456ms] [Info] [omni.meta-upgrade] [31794] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:fixVersioning():2091: Updated checkpoints ACLS: 0 2023-01-11 01:36:13 [2,471ms] [Info] [omni.core] [31794] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:initialize():585: Version: 114.4-84b6cd59 2023-01-11 01:36:13 [2,472ms] [Info] [omni.core] [31794] ../../../source/plugins/omni.core/impl/Mounts.cpp:createMount():97: Successfully created resolver of type: omniverse_resolver_s3 2023-01-11 01:36:13 [2,474ms] [Info] [omni.core] [31794] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:processPendingDeltaManagerRecords():1515: Checking for any pending delta manager records. 2023-01-11 01:36:13 [2,474ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():123: waiting for Core to startup ... 2023-01-11 01:36:13 [2,474ms] [Info] [omni.core] [31802] ../../../source/plugins/omni.core/impl/requests/DataStoreBase.cpp:runContentRemove():351: Started content remove task 2023-01-11 01:36:13 [2,474ms] [Info] [omni.core] [31803] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():599: Starting delta applier process. 2023-01-11 01:36:13 [2,474ms] [Info] [carb.launcher.plugin] [31803] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x7fdea40065e0 for './omni.delta-applier.app'. 2023-01-11 01:36:13 [2,474ms] [Info] [carb.launcher.plugin] [31803] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni.delta-applier.app' from process 31794. 2023-01-11 01:36:13 [2,476ms] [Info] [carb.launcher.plugin] [31810] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 32024 from the parent 31794. 2023-01-11 01:36:13 [2,476ms] [Info] [carb.launcher.plugin] [31810] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 32024. 2023-01-11 01:36:13 [2,477ms] [Info] [carb.launcher.plugin] [31810] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 32024. 2023-01-11 01:36:13 [2,477ms] [Info] [carb.launcher.plugin] [31803] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 32024 from the parent 31794. 2023-01-11 01:36:13 [2,477ms] [Info] [carb.launcher.plugin] [31803] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 32024 {process = 0x7fdea40065e0} 2023-01-11 01:36:13 [2,477ms] [Info] [carb.launcher.plugin] [32025] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 32024. 2023-01-11 01:36:13 [2,477ms] [Info] [carb.launcher.plugin] [32026] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 32024. 2023-01-11 01:36:13 [2,679ms] [Info] [omni.core] [31804] ../../../source/plugins/omni.core/impl/DeltaManager.cpp:pingTask():629: Successfully started delta applier process. 2023-01-11 01:36:15 [4,474ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():126: waiting for Core to startup (done) 2023-01-11 01:36:15 [4,485ms] [Info] [omni.server] [31794] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():137: Omniverse C++ ApiServer server started. 2023-01-11 01:36:15 [4,485ms] [Info] [omni.signalhandler] [31794] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():150: Installed signal handler for SIGINT 2023-01-11 01:36:15 [4,485ms] [Info] [omni.signalhandler] [31794] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():159: Installed signal handler for SIGTERM 2023-01-11 01:36:15 [4,485ms] [Info] [omni.signalhandler] [31794] ../../../source/plugins/omni.signalhandler/SignalHandler.cpp:SignalHandler():168: Ignoring SIGPIPE 2023-01-11 01:36:22 [12,004ms] [Info] [omni.core] [31804] ../../../source/plugins/common/ChildProcess.cpp:kill():125: Killing child server process ./omni-lft-service/omni-lft-service 2023-01-11 01:36:22 [12,004ms] [Info] [carb.launcher.plugin] [31804] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:kill():327: killing the child process 31811 {exitCode = 137, flags = 0x00000003, timeout = 5000} 2023-01-11 01:36:22 [12,005ms] [Info] [omni.core] [31804] ../../../source/plugins/common/ChildProcess.cpp:kill():127: Waiting for child server process ./omni-lft-service/omni-lft-service to exit 2023-01-11 01:36:22 [12,005ms] [Info] [carb.launcher.plugin] [31804] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():254: waiting for the process 0x1e5abb0 (31811) to exit {timeout = 18446744073709551615} 2023-01-11 01:36:22 [12,005ms] [Info] [carb.launcher.plugin] [31804] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:wait():259: the process has already exited with code 137. 2023-01-11 01:36:22 [12,005ms] [Info] [carb.launcher.plugin] [31804] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():247: destroying the process object 0x1e5abb0 {pid = 31811} 2023-01-11 01:36:22 [12,005ms] [Info] [carb.launcher.plugin] [31804] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:~ProcessImpl():249: done destroying the process object 0x1e5abb0 {pid = 31811} 2023-01-11 01:36:22 [12,005ms] [Info] [carb.launcher.plugin] [31804] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1e5abb0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:36:22 [12,005ms] [Info] [carb.launcher.plugin] [31804] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 31794. 2023-01-11 01:36:22 [12,008ms] [Info] [carb.launcher.plugin] [31810] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 32323 from the parent 31794. 2023-01-11 01:36:22 [12,008ms] [Info] [carb.launcher.plugin] [31810] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 32323. 2023-01-11 01:36:23 [12,014ms] [Info] [carb.launcher.plugin] [31810] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 32323. 2023-01-11 01:36:23 [12,014ms] [Info] [carb.launcher.plugin] [31804] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 32323 from the parent 31794. 2023-01-11 01:36:23 [12,014ms] [Info] [carb.launcher.plugin] [31804] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 32323 {process = 0x1e5abb0} 2023-01-11 01:36:23 [12,014ms] [Info] [carb.launcher.plugin] [32324] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 32323. 2023-01-11 01:36:23 [12,014ms] [Info] [omni.core] [31804] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:36:23 [12,014ms] [Info] [omni.core] [31804] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:36:23 [12,014ms] [Info] [carb.launcher.plugin] [32325] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 32323. 2023-01-11 01:36:23 [12,232ms] [Error] [omni.core] [32325] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:36:23 [12,235ms] [Error] [omni.core] [32325] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:36:23,227 - system_log - INFO - reconnecting ... 2023-01-11 01:36:23 [12,243ms] [Error] [omni.core] [32325] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:36:23,235 - system_log - INFO - connected 2023-01-11 01:36:23 [12,244ms] [Error] [omni.core] [32325] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Server started 2023-01-11 01:37:25 [0ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:37:25 [0ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:37:25 [0ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:37:25 [0ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:37:25 [0ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:37:25 [0ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:37:25 [0ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:37:25 [0ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:37:25 [0ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:37:25 [0ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:37:25 [1ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:37:25 [1ms] [Info] [omni.server] [978] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:37:25 [1ms] [Info] [omni.server] [978] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:37:25 [1ms] [Info] [omni.server] [978] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:37:25 [1ms] [Info] [omni.server] [978] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:37:25 [2ms] [Info] [omni.server] [978] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:37:25 [2ms] [Info] [omni.server] [978] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:37:25 [2ms] [Info] [omni.server] [978] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:37:25 [2ms] [Info] [omni.server] [978] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:37:25 [2ms] [Info] [omni.server] [978] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:37:25 [2ms] [Info] [omni.server] [978] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:37:25 [2ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:37:25 [2ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:37:25 [3ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:37:25 [3ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:37:25 [3ms] [Info] [carb.launcher.plugin] [978] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x180fad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:37:25 [3ms] [Info] [carb.launcher.plugin] [978] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 978. 2023-01-11 01:37:25 [3ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:37:25 [3ms] [Info] [carb] [978] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:37:25 [4ms] [Info] [carb.launcher.plugin] [990] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 991 from the parent 978. 2023-01-11 01:37:25 [4ms] [Info] [carb.launcher.plugin] [990] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 991. 2023-01-11 01:37:25 [4ms] [Info] [carb.launcher.plugin] [990] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 991. 2023-01-11 01:37:25 [4ms] [Info] [carb.launcher.plugin] [978] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 991 from the parent 978. 2023-01-11 01:37:25 [4ms] [Info] [carb.launcher.plugin] [978] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 991 {process = 0x180fad0} 2023-01-11 01:37:25 [4ms] [Info] [omni.core] [978] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:37:25 [4ms] [Info] [omni.core] [978] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:37:25 [4ms] [Info] [carb.launcher.plugin] [992] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 991. 2023-01-11 01:37:25 [4ms] [Info] [carb.launcher.plugin] [993] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 991. 2023-01-11 01:37:25 [225ms] [Error] [omni.core] [993] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:37:25 [228ms] [Error] [omni.core] [993] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:37:25,721 - system_log - INFO - reconnecting ... 2023-01-11 01:37:25 [236ms] [Error] [omni.core] [993] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:37:25,729 - system_log - INFO - connected 2023-01-11 01:37:25 [237ms] [Error] [omni.core] [993] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [991] 2023-01-11 01:37:25 [237ms] [Error] [omni.core] [993] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:37:25 [264ms] [Info] [carb.launcher.plugin] [992] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:37:25 [264ms] [Info] [carb.launcher.plugin] [993] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:37:25 [264ms] [Info] [carb.launcher.plugin] [992] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 991, bytes = 0} 2023-01-11 01:37:25 [264ms] [Info] [carb.launcher.plugin] [993] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 991, bytes = 0} 2023-01-11 01:37:25 [264ms] [Info] [carb.launcher.plugin] [992] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 991. 2023-01-11 01:37:25 [264ms] [Info] [carb.launcher.plugin] [993] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 991. 2023-01-11 01:37:27 [2,004ms] [Info] [omni.core] [984] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:37:27 [2,004ms] [Info] [omni.core] [984] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:37:27 [2,004ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:37:27 [2,005ms] [Info] [omni.core] [978] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:37:27 [2,005ms] [Info] [carb] [978] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:37:27 [2,005ms] [Warning] [omni.keyvaluedatabase.leveldb] [978] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:37:27 [2,005ms] [Info] [omni.core] [984] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:37:27 [2,059ms] [Warning] [omni.keyvaluedatabase.leveldb] [978] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:37:27 [2,059ms] [Fatal] [omni.core] [978] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-11 01:37:27 [0ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:37:27 [0ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:37:27 [0ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:37:27 [0ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:37:27 [0ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:37:27 [0ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:37:27 [1ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:37:27 [1ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:37:27 [1ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:37:27 [1ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:37:27 [1ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:37:27 [1ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:37:27 [1ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:37:27 [1ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:37:27 [1ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:37:27 [2ms] [Info] [carb] [1036] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:37:27 [2ms] [Info] [carb] [1036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:37:27 [2ms] [Info] [carb] [1036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:37:27 [2ms] [Info] [carb] [1036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:37:27 [2ms] [Info] [carb] [1036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:37:27 [2ms] [Info] [carb] [1036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:37:27 [2ms] [Info] [omni.server] [1036] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:37:27 [2ms] [Info] [omni.server] [1036] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:37:27 [2ms] [Info] [omni.server] [1036] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:37:27 [2ms] [Info] [omni.server] [1036] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:37:27 [2ms] [Info] [omni.server] [1036] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:37:27 [2ms] [Info] [omni.server] [1036] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:37:27 [2ms] [Info] [omni.server] [1036] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:37:27 [2ms] [Info] [omni.server] [1036] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:37:27 [2ms] [Info] [omni.server] [1036] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:37:27 [2ms] [Info] [omni.server] [1036] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:37:27 [2ms] [Info] [carb] [1036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:37:27 [3ms] [Info] [carb] [1036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:37:27 [3ms] [Info] [carb] [1036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:37:27 [3ms] [Info] [carb] [1036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:37:27 [3ms] [Info] [carb.launcher.plugin] [1036] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x23c8ac0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:37:27 [3ms] [Info] [carb.launcher.plugin] [1036] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 1036. 2023-01-11 01:37:27 [3ms] [Info] [carb] [1036] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:37:27 [3ms] [Info] [carb] [1036] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:37:27 [3ms] [Info] [carb.launcher.plugin] [1057] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 1058 from the parent 1036. 2023-01-11 01:37:27 [4ms] [Info] [carb.launcher.plugin] [1057] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 1058. 2023-01-11 01:37:27 [4ms] [Info] [carb.launcher.plugin] [1057] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 1058. 2023-01-11 01:37:27 [4ms] [Info] [carb.launcher.plugin] [1036] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 1058 from the parent 1036. 2023-01-11 01:37:27 [4ms] [Info] [carb.launcher.plugin] [1036] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 1058 {process = 0x23c8ac0} 2023-01-11 01:37:27 [4ms] [Info] [omni.core] [1036] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:37:27 [4ms] [Info] [omni.core] [1036] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:37:27 [4ms] [Info] [carb.launcher.plugin] [1060] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 1058. 2023-01-11 01:37:27 [4ms] [Info] [carb.launcher.plugin] [1061] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 1058. 2023-01-11 01:37:27 [231ms] [Error] [omni.core] [1061] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:37:27 [234ms] [Error] [omni.core] [1061] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:37:27,949 - system_log - INFO - reconnecting ... 2023-01-11 01:37:27 [242ms] [Error] [omni.core] [1061] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:37:27,957 - system_log - INFO - connected 2023-01-11 01:37:27 [243ms] [Error] [omni.core] [1061] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [1058] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:37:27 [270ms] [Info] [carb.launcher.plugin] [1061] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:37:27 [270ms] [Info] [carb.launcher.plugin] [1060] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:37:27 [270ms] [Info] [carb.launcher.plugin] [1061] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 1058, bytes = 0} 2023-01-11 01:37:27 [270ms] [Info] [carb.launcher.plugin] [1060] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 1058, bytes = 0} 2023-01-11 01:37:27 [270ms] [Info] [carb.launcher.plugin] [1061] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 1058. 2023-01-11 01:37:27 [270ms] [Info] [carb.launcher.plugin] [1060] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 1058. 2023-01-11 01:37:29 [0ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:37:29 [0ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:37:29 [0ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:37:29 [0ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:37:29 [0ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:37:29 [0ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:37:29 [0ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:37:29 [0ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:37:29 [0ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:37:29 [0ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:37:29 [1ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:37:29 [1ms] [Info] [omni.server] [1082] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:37:29 [1ms] [Info] [omni.server] [1082] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:37:29 [1ms] [Info] [omni.server] [1082] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:37:29 [1ms] [Info] [omni.server] [1082] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:37:29 [1ms] [Info] [omni.server] [1082] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:37:29 [1ms] [Info] [omni.server] [1082] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:37:29 [1ms] [Info] [omni.server] [1082] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:37:29 [1ms] [Info] [omni.server] [1082] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:37:29 [1ms] [Info] [omni.server] [1082] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:37:29 [2ms] [Info] [omni.server] [1082] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:37:29 [2ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:37:29 [2ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:37:29 [2ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:37:29 [2ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:37:29 [3ms] [Info] [carb.launcher.plugin] [1082] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1324ad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:37:29 [3ms] [Info] [carb.launcher.plugin] [1082] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 1082. 2023-01-11 01:37:29 [3ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:37:29 [3ms] [Info] [carb] [1082] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:37:29 [3ms] [Info] [carb.launcher.plugin] [1097] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 1101 from the parent 1082. 2023-01-11 01:37:29 [3ms] [Info] [carb.launcher.plugin] [1097] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 1101. 2023-01-11 01:37:29 [4ms] [Info] [carb.launcher.plugin] [1097] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 1101. 2023-01-11 01:37:29 [4ms] [Info] [carb.launcher.plugin] [1082] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 1101 from the parent 1082. 2023-01-11 01:37:29 [4ms] [Info] [carb.launcher.plugin] [1082] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 1101 {process = 0x1324ad0} 2023-01-11 01:37:29 [4ms] [Info] [omni.core] [1082] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:37:29 [4ms] [Info] [omni.core] [1082] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:37:29 [4ms] [Info] [carb.launcher.plugin] [1103] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 1101. 2023-01-11 01:37:29 [4ms] [Info] [carb.launcher.plugin] [1102] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 1101. 2023-01-11 01:37:29 [234ms] [Error] [omni.core] [1103] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:37:29 [237ms] [Error] [omni.core] [1103] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:37:29,387 - system_log - INFO - reconnecting ... 2023-01-11 01:37:29 [245ms] [Error] [omni.core] [1103] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:37:29,395 - system_log - INFO - connected 2023-01-11 01:37:29 [246ms] [Error] [omni.core] [1103] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use 2023-01-11 01:37:29 [246ms] [Error] [omni.core] [1103] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:[1101] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:37:29 [274ms] [Info] [carb.launcher.plugin] [1102] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:37:29 [274ms] [Info] [carb.launcher.plugin] [1102] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 1101, bytes = 0} 2023-01-11 01:37:29 [274ms] [Info] [carb.launcher.plugin] [1102] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 1101. 2023-01-11 01:37:29 [274ms] [Info] [carb.launcher.plugin] [1103] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:37:29 [274ms] [Info] [carb.launcher.plugin] [1103] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 1101, bytes = 0} 2023-01-11 01:37:29 [274ms] [Info] [carb.launcher.plugin] [1103] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 1101. 2023-01-11 01:37:31 [2,004ms] [Info] [omni.core] [1087] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:37:31 [2,004ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:37:31 [2,004ms] [Info] [omni.core] [1087] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:37:31 [2,004ms] [Info] [omni.core] [1082] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:37:31 [2,004ms] [Info] [carb] [1082] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:37:31 [2,005ms] [Warning] [omni.keyvaluedatabase.leveldb] [1082] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:37:31 [2,005ms] [Info] [omni.core] [1087] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:37:31 [2,022ms] [Warning] [omni.keyvaluedatabase.leveldb] [1082] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:37:31 [2,022ms] [Fatal] [omni.core] [1082] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4900] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4900] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4900] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4900] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4900] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4900] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4900] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4900] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4900] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4900] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4900] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4900] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4900] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4900] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4900] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:40:56 [2ms] [Info] [omni.server] [4900] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:40:56 [2ms] [Info] [carb] [4900] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:40:56 [2ms] [Info] [carb] [4900] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:40:56 [2ms] [Info] [carb] [4900] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:40:56 [3ms] [Info] [carb] [4900] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4900] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0xa1fad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4900] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 4900. 2023-01-11 01:40:56 [3ms] [Info] [carb] [4900] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:40:56 [3ms] [Info] [carb] [4900] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4912] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4913 from the parent 4900. 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4912] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4913. 2023-01-11 01:40:56 [4ms] [Info] [carb.launcher.plugin] [4912] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4913. 2023-01-11 01:40:56 [4ms] [Info] [carb.launcher.plugin] [4900] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4913 from the parent 4900. 2023-01-11 01:40:56 [4ms] [Info] [carb.launcher.plugin] [4900] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4913 {process = 0xa1fad0} 2023-01-11 01:40:56 [4ms] [Info] [omni.core] [4900] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:40:56 [4ms] [Info] [carb.launcher.plugin] [4914] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4913. 2023-01-11 01:40:56 [4ms] [Info] [omni.core] [4900] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:40:56 [4ms] [Info] [carb.launcher.plugin] [4915] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4913. 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:40:56 [0ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4916] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:40:56 [1ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4916] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4916] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4916] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4916] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4916] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4916] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4916] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4916] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4916] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:40:56 [1ms] [Info] [omni.server] [4916] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:40:56 [2ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:40:56 [2ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:40:56 [2ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:40:56 [2ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:40:56 [2ms] [Info] [carb.launcher.plugin] [4916] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1a36a50 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:40:56 [2ms] [Info] [carb.launcher.plugin] [4916] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 4916. 2023-01-11 01:40:56 [2ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:40:56 [2ms] [Info] [carb] [4916] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4928] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 4929 from the parent 4916. 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4928] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 4929. 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4928] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 4929. 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4916] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 4929 from the parent 4916. 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4916] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 4929 {process = 0x1a36a50} 2023-01-11 01:40:56 [3ms] [Info] [omni.core] [4916] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:40:56 [3ms] [Info] [omni.core] [4916] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4931] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 4929. 2023-01-11 01:40:56 [3ms] [Info] [carb.launcher.plugin] [4930] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 4929. 2023-01-11 01:40:56 [232ms] [Error] [omni.core] [4931] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:40:56 [236ms] [Error] [omni.core] [4931] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:40:56,398 - system_log - INFO - reconnecting ... 2023-01-11 01:40:56 [244ms] [Error] [omni.core] [4931] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:40:56,406 - system_log - INFO - connected 2023-01-11 01:40:56 [244ms] [Error] [omni.core] [4931] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [4929] 2023-01-11 01:40:56 [244ms] [Error] [omni.core] [4931] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:40:56 [268ms] [Info] [carb.launcher.plugin] [4930] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:40:56 [268ms] [Info] [carb.launcher.plugin] [4931] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:40:56 [268ms] [Info] [carb.launcher.plugin] [4930] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 4929, bytes = 0} 2023-01-11 01:40:56 [268ms] [Info] [carb.launcher.plugin] [4931] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 4929, bytes = 0} 2023-01-11 01:40:56 [268ms] [Info] [carb.launcher.plugin] [4930] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 4929. 2023-01-11 01:40:56 [268ms] [Info] [carb.launcher.plugin] [4931] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 4929. 2023-01-11 01:40:58 [2,004ms] [Info] [omni.core] [4921] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:40:58 [2,004ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:40:58 [2,004ms] [Info] [omni.core] [4921] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:40:58 [2,006ms] [Info] [omni.core] [4916] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:40:58 [2,006ms] [Info] [carb] [4916] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:40:58 [2,007ms] [Warning] [omni.keyvaluedatabase.leveldb] [4916] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:40:58 [2,008ms] [Info] [omni.core] [4921] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:40:58 [2,048ms] [Warning] [omni.keyvaluedatabase.leveldb] [4916] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:40:58 [2,048ms] [Fatal] [omni.core] [4916] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-11 01:41:01 [0ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5028] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5028] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5028] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5028] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5028] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5028] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5028] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5028] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5028] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5028] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5028] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:41:01 [3ms] [Info] [carb] [5028] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:41:01 [3ms] [Info] [carb] [5028] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:41:01 [3ms] [Info] [carb] [5028] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:41:01 [4ms] [Info] [carb] [5028] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:41:01 [4ms] [Info] [carb.launcher.plugin] [5028] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x26d1ad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:41:01 [4ms] [Info] [carb.launcher.plugin] [5028] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 5028. 2023-01-11 01:41:01 [4ms] [Info] [carb] [5028] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:41:01 [4ms] [Info] [carb] [5028] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:41:01 [4ms] [Info] [carb.launcher.plugin] [5040] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 5041 from the parent 5028. 2023-01-11 01:41:01 [4ms] [Info] [carb.launcher.plugin] [5040] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 5041. 2023-01-11 01:41:01 [5ms] [Info] [carb.launcher.plugin] [5040] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 5041. 2023-01-11 01:41:01 [5ms] [Info] [carb.launcher.plugin] [5028] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 5041 from the parent 5028. 2023-01-11 01:41:01 [5ms] [Info] [carb.launcher.plugin] [5028] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 5041 {process = 0x26d1ad0} 2023-01-11 01:41:01 [5ms] [Info] [omni.core] [5028] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:41:01 [5ms] [Info] [carb.launcher.plugin] [5043] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 5041. 2023-01-11 01:41:01 [5ms] [Info] [omni.core] [5028] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:41:01 [5ms] [Info] [carb.launcher.plugin] [5042] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 5041. 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5046] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5046] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5046] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5046] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5046] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5046] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5046] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5046] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5046] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5046] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:41:01 [1ms] [Info] [carb] [5046] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:41:01 [2ms] [Info] [carb] [5046] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:41:01 [2ms] [Info] [carb] [5046] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:41:01 [2ms] [Info] [carb] [5046] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:41:01 [2ms] [Info] [carb.launcher.plugin] [5046] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x19dca50 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:41:01 [2ms] [Info] [carb.launcher.plugin] [5046] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 5046. 2023-01-11 01:41:01 [2ms] [Info] [carb] [5046] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:41:01 [2ms] [Info] [carb] [5046] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5062] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 5063 from the parent 5046. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5062] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 5063. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5062] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 5063. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5046] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 5063 from the parent 5046. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5046] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 5063 {process = 0x19dca50} 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5064] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 5063. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5065] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 5063. 2023-01-11 01:41:01 [3ms] [Info] [omni.core] [5046] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:41:01 [3ms] [Info] [omni.core] [5046] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:41:01 [212ms] [Error] [omni.core] [5065] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:41:01 [215ms] [Error] [omni.core] [5065] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:41:01,489 - system_log - INFO - reconnecting ... 2023-01-11 01:41:01 [223ms] [Error] [omni.core] [5065] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:41:01,496 - system_log - INFO - connected 2023-01-11 01:41:01 [223ms] [Error] [omni.core] [5065] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [5063] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:41:01 [245ms] [Info] [carb.launcher.plugin] [5064] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:41:01 [245ms] [Info] [carb.launcher.plugin] [5064] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 5063, bytes = 0} 2023-01-11 01:41:01 [245ms] [Info] [carb.launcher.plugin] [5064] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 5063. 2023-01-11 01:41:01 [245ms] [Info] [carb.launcher.plugin] [5065] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:41:01 [245ms] [Info] [carb.launcher.plugin] [5065] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 5063, bytes = 0} 2023-01-11 01:41:01 [245ms] [Info] [carb.launcher.plugin] [5065] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 5063. 2023-01-11 01:41:01 [0ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:41:01 [0ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:41:01 [1ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5095] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5095] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5095] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5095] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5095] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5095] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5095] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5095] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:41:01 [1ms] [Info] [omni.server] [5095] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:41:01 [2ms] [Info] [omni.server] [5095] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:41:01 [2ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:41:01 [2ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:41:01 [2ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:41:01 [2ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:41:01 [2ms] [Info] [carb.launcher.plugin] [5095] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1067ad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:41:01 [2ms] [Info] [carb.launcher.plugin] [5095] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 5095. 2023-01-11 01:41:01 [2ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:41:01 [2ms] [Info] [carb] [5095] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5114] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 5115 from the parent 5095. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5114] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 5115. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5114] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 5115. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5095] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 5115 from the parent 5095. 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5095] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 5115 {process = 0x1067ad0} 2023-01-11 01:41:01 [3ms] [Info] [omni.core] [5095] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:41:01 [3ms] [Info] [omni.core] [5095] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:41:01 [3ms] [Info] [carb.launcher.plugin] [5116] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 5115. 2023-01-11 01:41:01 [4ms] [Info] [carb.launcher.plugin] [5117] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 5115. 2023-01-11 01:41:02 [218ms] [Error] [omni.core] [5117] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:41:02 [221ms] [Error] [omni.core] [5117] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:41:02,120 - system_log - INFO - reconnecting ... 2023-01-11 01:41:02 [229ms] [Error] [omni.core] [5117] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:41:02,128 - system_log - INFO - connected 2023-01-11 01:41:02 [229ms] [Error] [omni.core] [5117] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [5115] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:41:02 [254ms] [Info] [carb.launcher.plugin] [5116] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:41:02 [254ms] [Info] [carb.launcher.plugin] [5116] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 5115, bytes = 0} 2023-01-11 01:41:02 [255ms] [Info] [carb.launcher.plugin] [5116] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 5115. 2023-01-11 01:41:02 [254ms] [Info] [carb.launcher.plugin] [5117] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:41:02 [255ms] [Info] [carb.launcher.plugin] [5117] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 5115, bytes = 0} 2023-01-11 01:41:02 [255ms] [Info] [carb.launcher.plugin] [5117] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 5115. 2023-01-11 01:41:03 [2,004ms] [Info] [omni.core] [5109] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:41:03 [2,004ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:41:03 [2,004ms] [Info] [omni.core] [5109] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:41:03 [2,006ms] [Info] [omni.core] [5095] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:41:03 [2,007ms] [Info] [carb] [5095] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:41:03 [2,007ms] [Warning] [omni.keyvaluedatabase.leveldb] [5095] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:41:03 [2,008ms] [Info] [omni.core] [5109] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:41:03 [2,027ms] [Warning] [omni.keyvaluedatabase.leveldb] [5095] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:41:03 [2,027ms] [Fatal] [omni.core] [5095] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-11 01:56:24 [0ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:56:24 [0ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:56:24 [0ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:56:24 [0ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:56:24 [0ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:56:24 [0ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:56:24 [0ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:56:24 [0ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:56:24 [1ms] [Info] [carb] [16616] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:56:24 [15ms] [Info] [omni.server] [16616] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:56:24 [15ms] [Info] [omni.server] [16616] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:56:24 [15ms] [Info] [omni.server] [16616] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:56:24 [15ms] [Info] [omni.server] [16616] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:56:24 [15ms] [Info] [omni.server] [16616] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:56:24 [15ms] [Info] [omni.server] [16616] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:56:24 [15ms] [Info] [omni.server] [16616] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:56:24 [15ms] [Info] [omni.server] [16616] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:56:24 [15ms] [Info] [omni.server] [16616] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:56:24 [16ms] [Info] [omni.server] [16616] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:56:24 [16ms] [Info] [carb] [16616] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:56:24 [27ms] [Info] [carb] [16616] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:56:24 [28ms] [Info] [carb] [16616] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:56:24 [28ms] [Info] [carb] [16616] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:56:24 [28ms] [Info] [carb.launcher.plugin] [16616] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x2674ba0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:56:24 [28ms] [Info] [carb.launcher.plugin] [16616] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 16616. 2023-01-11 01:56:24 [28ms] [Info] [carb] [16616] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:56:24 [28ms] [Info] [carb] [16616] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:56:24 [29ms] [Info] [carb.launcher.plugin] [16635] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 16638 from the parent 16616. 2023-01-11 01:56:24 [29ms] [Info] [carb.launcher.plugin] [16635] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 16638. 2023-01-11 01:56:24 [30ms] [Info] [carb.launcher.plugin] [16635] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 16638. 2023-01-11 01:56:24 [30ms] [Info] [carb.launcher.plugin] [16616] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 16638 from the parent 16616. 2023-01-11 01:56:24 [30ms] [Info] [carb.launcher.plugin] [16616] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 16638 {process = 0x2674ba0} 2023-01-11 01:56:24 [30ms] [Info] [carb.launcher.plugin] [16639] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 16638. 2023-01-11 01:56:24 [30ms] [Info] [omni.core] [16616] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:56:24 [30ms] [Info] [omni.core] [16616] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:56:24 [39ms] [Info] [carb.launcher.plugin] [16640] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 16638. 2023-01-11 01:56:25 [342ms] [Error] [omni.core] [16640] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:56:25 [346ms] [Error] [omni.core] [16640] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:56:25,071 - system_log - INFO - reconnecting ... 2023-01-11 01:56:25 [357ms] [Error] [omni.core] [16640] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:56:25,082 - system_log - INFO - connected 2023-01-11 01:56:25 [357ms] [Error] [omni.core] [16640] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [16638] 2023-01-11 01:56:25 [357ms] [Error] [omni.core] [16640] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:56:25 [391ms] [Info] [carb.launcher.plugin] [16639] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:56:25 [391ms] [Info] [carb.launcher.plugin] [16640] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:56:25 [391ms] [Info] [carb.launcher.plugin] [16640] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 16638, bytes = 0} 2023-01-11 01:56:25 [391ms] [Info] [carb.launcher.plugin] [16639] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 16638, bytes = 0} 2023-01-11 01:56:25 [391ms] [Info] [carb.launcher.plugin] [16640] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 16638. 2023-01-11 01:56:25 [391ms] [Info] [carb.launcher.plugin] [16639] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 16638. 2023-01-11 01:56:39 [0ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:56:39 [0ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:56:39 [0ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:56:39 [0ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:56:39 [0ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:56:39 [0ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:56:39 [0ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:56:39 [1ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:56:39 [1ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:56:39 [1ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:56:39 [1ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:56:39 [1ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:56:39 [1ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:56:39 [1ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:56:39 [1ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:56:39 [2ms] [Info] [carb] [17299] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:56:39 [2ms] [Info] [carb] [17299] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:56:39 [2ms] [Info] [carb] [17299] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:56:39 [2ms] [Info] [carb] [17299] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:56:39 [2ms] [Info] [carb] [17299] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:56:39 [2ms] [Info] [carb] [17299] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:56:39 [21ms] [Info] [omni.server] [17299] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:56:39 [21ms] [Info] [omni.server] [17299] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:56:39 [21ms] [Info] [omni.server] [17299] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:56:39 [21ms] [Info] [omni.server] [17299] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:56:39 [21ms] [Info] [omni.server] [17299] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:56:39 [21ms] [Info] [omni.server] [17299] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:56:39 [21ms] [Info] [omni.server] [17299] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:56:39 [21ms] [Info] [omni.server] [17299] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:56:39 [21ms] [Info] [omni.server] [17299] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:56:39 [22ms] [Info] [omni.server] [17299] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:56:39 [22ms] [Info] [carb] [17299] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:56:39 [22ms] [Info] [carb] [17299] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:56:39 [23ms] [Info] [carb] [17299] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:56:39 [24ms] [Info] [carb] [17299] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:56:39 [24ms] [Info] [carb.launcher.plugin] [17299] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x23a3bb0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:56:39 [24ms] [Info] [carb.launcher.plugin] [17299] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 17299. 2023-01-11 01:56:39 [24ms] [Info] [carb] [17299] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:56:39 [24ms] [Info] [carb] [17299] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:56:39 [38ms] [Info] [carb.launcher.plugin] [17317] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 17319 from the parent 17299. 2023-01-11 01:56:39 [38ms] [Info] [carb.launcher.plugin] [17317] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 17319. 2023-01-11 01:56:39 [39ms] [Info] [carb.launcher.plugin] [17317] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 17319. 2023-01-11 01:56:39 [39ms] [Info] [carb.launcher.plugin] [17299] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 17319 from the parent 17299. 2023-01-11 01:56:39 [39ms] [Info] [carb.launcher.plugin] [17299] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 17319 {process = 0x23a3bb0} 2023-01-11 01:56:39 [39ms] [Info] [omni.core] [17299] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:56:39 [39ms] [Info] [omni.core] [17299] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:56:39 [39ms] [Info] [carb.launcher.plugin] [17322] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 17319. 2023-01-11 01:56:39 [39ms] [Info] [carb.launcher.plugin] [17321] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 17319. 2023-01-11 01:56:39 [382ms] [Error] [omni.core] [17322] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:56:39 [386ms] [Error] [omni.core] [17322] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:56:39,907 - system_log - INFO - reconnecting ... 2023-01-11 01:56:39 [395ms] [Error] [omni.core] [17322] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:56:39,916 - system_log - INFO - connected 2023-01-11 01:56:39 [395ms] [Error] [omni.core] [17322] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [17319] 2023-01-11 01:56:39 [395ms] [Error] [omni.core] [17322] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:56:39 [434ms] [Info] [carb.launcher.plugin] [17321] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:56:39 [434ms] [Info] [carb.launcher.plugin] [17322] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:56:39 [434ms] [Info] [carb.launcher.plugin] [17322] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 17319, bytes = 0} 2023-01-11 01:56:39 [434ms] [Info] [carb.launcher.plugin] [17321] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 17319, bytes = 0} 2023-01-11 01:56:39 [434ms] [Info] [carb.launcher.plugin] [17322] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 17319. 2023-01-11 01:56:39 [434ms] [Info] [carb.launcher.plugin] [17321] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 17319. 2023-01-11 01:57:33 [0ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:57:33 [0ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:57:33 [0ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:33 [0ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:33 [0ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:33 [0ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:57:33 [0ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:33 [0ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:57:33 [0ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:33 [0ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:33 [1ms] [Info] [carb] [18648] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:33 [1ms] [Info] [omni.server] [18648] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:57:33 [1ms] [Info] [omni.server] [18648] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:57:33 [1ms] [Info] [omni.server] [18648] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:57:33 [1ms] [Info] [omni.server] [18648] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:57:33 [1ms] [Info] [omni.server] [18648] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:57:33 [1ms] [Info] [omni.server] [18648] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:57:33 [1ms] [Info] [omni.server] [18648] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:57:33 [1ms] [Info] [omni.server] [18648] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:57:33 [1ms] [Info] [omni.server] [18648] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:57:33 [2ms] [Info] [omni.server] [18648] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:57:33 [2ms] [Info] [carb] [18648] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:33 [2ms] [Info] [carb] [18648] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:33 [2ms] [Info] [carb] [18648] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:33 [3ms] [Info] [carb] [18648] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:33 [3ms] [Info] [carb.launcher.plugin] [18648] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x19baae0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:57:33 [3ms] [Info] [carb.launcher.plugin] [18648] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 18648. 2023-01-11 01:57:33 [3ms] [Info] [carb] [18648] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:57:33 [3ms] [Info] [carb] [18648] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:57:33 [3ms] [Info] [carb.launcher.plugin] [18660] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 18661 from the parent 18648. 2023-01-11 01:57:33 [3ms] [Info] [carb.launcher.plugin] [18660] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 18661. 2023-01-11 01:57:33 [4ms] [Info] [carb.launcher.plugin] [18660] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 18661. 2023-01-11 01:57:33 [4ms] [Info] [carb.launcher.plugin] [18648] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 18661 from the parent 18648. 2023-01-11 01:57:33 [4ms] [Info] [carb.launcher.plugin] [18648] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 18661 {process = 0x19baae0} 2023-01-11 01:57:33 [4ms] [Info] [omni.core] [18648] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:57:33 [4ms] [Info] [omni.core] [18648] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:57:33 [4ms] [Info] [carb.launcher.plugin] [18662] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 18661. 2023-01-11 01:57:33 [4ms] [Info] [carb.launcher.plugin] [18663] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 18661. 2023-01-11 01:57:34 [233ms] [Error] [omni.core] [18663] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:57:34 [236ms] [Error] [omni.core] [18663] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:34,125 - system_log - INFO - reconnecting ... 2023-01-11 01:57:34 [245ms] [Error] [omni.core] [18663] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:34,134 - system_log - INFO - connected 2023-01-11 01:57:34 [245ms] [Error] [omni.core] [18663] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [18661] 2023-01-11 01:57:34 [245ms] [Error] [omni.core] [18663] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:57:34 [274ms] [Info] [carb.launcher.plugin] [18662] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:34 [274ms] [Info] [carb.launcher.plugin] [18662] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 18661, bytes = 0} 2023-01-11 01:57:34 [274ms] [Info] [carb.launcher.plugin] [18662] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 18661. 2023-01-11 01:57:34 [274ms] [Info] [carb.launcher.plugin] [18663] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:34 [274ms] [Info] [carb.launcher.plugin] [18663] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 18661, bytes = 0} 2023-01-11 01:57:34 [274ms] [Info] [carb.launcher.plugin] [18663] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 18661. 2023-01-11 01:57:35 [0ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:57:35 [0ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:57:35 [0ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:35 [0ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:35 [0ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:35 [0ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:57:35 [0ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:35 [0ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:57:35 [0ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:35 [1ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:35 [2ms] [Info] [omni.server] [18699] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:57:35 [2ms] [Info] [omni.server] [18699] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:57:35 [2ms] [Info] [omni.server] [18699] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:57:35 [2ms] [Info] [omni.server] [18699] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:57:35 [2ms] [Info] [omni.server] [18699] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:57:35 [2ms] [Info] [omni.server] [18699] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:57:35 [2ms] [Info] [omni.server] [18699] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:57:35 [2ms] [Info] [omni.server] [18699] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:57:35 [2ms] [Info] [omni.server] [18699] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:57:35 [2ms] [Info] [omni.server] [18699] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:57:35 [2ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:35 [2ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:35 [3ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:35 [3ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:35 [3ms] [Info] [carb.launcher.plugin] [18699] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x19f8ad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:57:35 [3ms] [Info] [carb.launcher.plugin] [18699] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 18699. 2023-01-11 01:57:35 [3ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:57:35 [3ms] [Info] [carb] [18699] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:57:35 [3ms] [Info] [carb.launcher.plugin] [18711] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 18712 from the parent 18699. 2023-01-11 01:57:35 [3ms] [Info] [carb.launcher.plugin] [18711] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 18712. 2023-01-11 01:57:35 [4ms] [Info] [carb.launcher.plugin] [18711] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 18712. 2023-01-11 01:57:35 [4ms] [Info] [carb.launcher.plugin] [18699] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 18712 from the parent 18699. 2023-01-11 01:57:35 [4ms] [Info] [carb.launcher.plugin] [18699] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 18712 {process = 0x19f8ad0} 2023-01-11 01:57:35 [4ms] [Info] [omni.core] [18699] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:57:35 [4ms] [Info] [omni.core] [18699] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:57:35 [4ms] [Info] [carb.launcher.plugin] [18713] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 18712. 2023-01-11 01:57:35 [4ms] [Info] [carb.launcher.plugin] [18714] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 18712. 2023-01-11 01:57:35 [243ms] [Error] [omni.core] [18714] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:57:35 [246ms] [Error] [omni.core] [18714] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:35,977 - system_log - INFO - reconnecting ... 2023-01-11 01:57:35 [254ms] [Error] [omni.core] [18714] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:35,985 - system_log - INFO - connected 2023-01-11 01:57:35 [255ms] [Error] [omni.core] [18714] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [18712] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:57:36 [281ms] [Info] [carb.launcher.plugin] [18714] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:36 [281ms] [Info] [carb.launcher.plugin] [18713] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:36 [281ms] [Info] [carb.launcher.plugin] [18714] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 18712, bytes = 0} 2023-01-11 01:57:36 [282ms] [Info] [carb.launcher.plugin] [18713] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 18712, bytes = 0} 2023-01-11 01:57:36 [282ms] [Info] [carb.launcher.plugin] [18714] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 18712. 2023-01-11 01:57:36 [282ms] [Info] [carb.launcher.plugin] [18713] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 18712. 2023-01-11 01:57:37 [2,005ms] [Info] [omni.core] [18704] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:57:37 [2,005ms] [Info] [omni.core] [18704] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:57:37 [2,005ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:57:37 [2,005ms] [Info] [omni.core] [18699] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:57:37 [2,005ms] [Info] [carb] [18699] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:57:37 [2,005ms] [Warning] [omni.keyvaluedatabase.leveldb] [18699] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:57:37 [2,005ms] [Info] [omni.core] [18704] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:57:37 [2,023ms] [Warning] [omni.keyvaluedatabase.leveldb] [18699] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:57:37 [2,023ms] [Fatal] [omni.core] [18699] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-11 01:57:39 [0ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:57:39 [0ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:57:39 [0ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:39 [0ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:39 [0ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:39 [0ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:57:39 [0ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:39 [1ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:39 [2ms] [Info] [omni.server] [18792] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:57:39 [2ms] [Info] [omni.server] [18792] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:57:39 [2ms] [Info] [omni.server] [18792] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:57:39 [2ms] [Info] [omni.server] [18792] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:57:39 [2ms] [Info] [omni.server] [18792] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:57:39 [2ms] [Info] [omni.server] [18792] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:57:39 [2ms] [Info] [omni.server] [18792] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:57:39 [2ms] [Info] [omni.server] [18792] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:57:39 [2ms] [Info] [omni.server] [18792] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:57:39 [2ms] [Info] [omni.server] [18792] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:57:39 [2ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:39 [2ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:39 [3ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:39 [3ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:39 [3ms] [Info] [carb.launcher.plugin] [18792] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x2783ad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:57:39 [3ms] [Info] [carb.launcher.plugin] [18792] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 18792. 2023-01-11 01:57:39 [3ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:57:39 [3ms] [Info] [carb] [18792] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:57:39 [3ms] [Info] [carb.launcher.plugin] [18804] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 18805 from the parent 18792. 2023-01-11 01:57:39 [3ms] [Info] [carb.launcher.plugin] [18804] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 18805. 2023-01-11 01:57:39 [4ms] [Info] [carb.launcher.plugin] [18804] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 18805. 2023-01-11 01:57:39 [4ms] [Info] [carb.launcher.plugin] [18792] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 18805 from the parent 18792. 2023-01-11 01:57:39 [4ms] [Info] [carb.launcher.plugin] [18792] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 18805 {process = 0x2783ad0} 2023-01-11 01:57:39 [4ms] [Info] [carb.launcher.plugin] [18806] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 18805. 2023-01-11 01:57:39 [4ms] [Info] [omni.core] [18792] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:57:39 [4ms] [Info] [omni.core] [18792] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:57:39 [4ms] [Info] [carb.launcher.plugin] [18807] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 18805. 2023-01-11 01:57:39 [226ms] [Error] [omni.core] [18807] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:57:39 [229ms] [Error] [omni.core] [18807] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:39,379 - system_log - INFO - reconnecting ... 2023-01-11 01:57:39 [237ms] [Error] [omni.core] [18807] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:39,387 - system_log - INFO - connected 2023-01-11 01:57:39 [237ms] [Error] [omni.core] [18807] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use 2023-01-11 01:57:39 [237ms] [Error] [omni.core] [18807] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:[18805] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:57:39 [264ms] [Info] [carb.launcher.plugin] [18807] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:39 [264ms] [Info] [carb.launcher.plugin] [18806] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:39 [264ms] [Info] [carb.launcher.plugin] [18807] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 18805, bytes = 0} 2023-01-11 01:57:39 [264ms] [Info] [carb.launcher.plugin] [18807] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 18805. 2023-01-11 01:57:39 [264ms] [Info] [carb.launcher.plugin] [18806] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 18805, bytes = 0} 2023-01-11 01:57:39 [264ms] [Info] [carb.launcher.plugin] [18806] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 18805. 2023-01-11 01:57:41 [2,005ms] [Info] [omni.core] [18799] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:57:41 [2,005ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:57:41 [2,005ms] [Info] [omni.core] [18799] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:57:41 [2,007ms] [Info] [omni.core] [18792] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:57:41 [2,007ms] [Info] [carb] [18792] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:57:41 [2,008ms] [Warning] [omni.keyvaluedatabase.leveldb] [18792] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:57:41 [2,008ms] [Info] [omni.core] [18799] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:57:41 [2,037ms] [Warning] [omni.keyvaluedatabase.leveldb] [18792] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:57:41 [2,037ms] [Fatal] [omni.core] [18792] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-11 01:57:42 [0ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:57:42 [0ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:57:42 [0ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:42 [0ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:42 [0ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:42 [0ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:57:42 [0ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:42 [1ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:42 [2ms] [Info] [omni.server] [18867] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:57:42 [2ms] [Info] [omni.server] [18867] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:57:42 [2ms] [Info] [omni.server] [18867] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:57:42 [2ms] [Info] [omni.server] [18867] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:57:42 [2ms] [Info] [omni.server] [18867] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:57:42 [2ms] [Info] [omni.server] [18867] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:57:42 [2ms] [Info] [omni.server] [18867] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:57:42 [2ms] [Info] [omni.server] [18867] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:57:42 [2ms] [Info] [omni.server] [18867] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:57:42 [2ms] [Info] [omni.server] [18867] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:57:42 [2ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:42 [2ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:42 [2ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:42 [3ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:42 [3ms] [Info] [carb.launcher.plugin] [18867] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1edeac0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:57:42 [3ms] [Info] [carb.launcher.plugin] [18867] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 18867. 2023-01-11 01:57:42 [3ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:57:42 [3ms] [Info] [carb] [18867] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:57:42 [3ms] [Info] [carb.launcher.plugin] [18879] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 18880 from the parent 18867. 2023-01-11 01:57:42 [3ms] [Info] [carb.launcher.plugin] [18879] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 18880. 2023-01-11 01:57:42 [4ms] [Info] [carb.launcher.plugin] [18879] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 18880. 2023-01-11 01:57:42 [4ms] [Info] [carb.launcher.plugin] [18867] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 18880 from the parent 18867. 2023-01-11 01:57:42 [4ms] [Info] [carb.launcher.plugin] [18867] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 18880 {process = 0x1edeac0} 2023-01-11 01:57:42 [4ms] [Info] [omni.core] [18867] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:57:42 [4ms] [Info] [omni.core] [18867] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:57:42 [4ms] [Info] [carb.launcher.plugin] [18882] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 18880. 2023-01-11 01:57:42 [4ms] [Info] [carb.launcher.plugin] [18881] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 18880. 2023-01-11 01:57:42 [225ms] [Error] [omni.core] [18882] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:57:42 [228ms] [Error] [omni.core] [18882] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:42,760 - system_log - INFO - reconnecting ... 2023-01-11 01:57:42 [236ms] [Error] [omni.core] [18882] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:42,768 - system_log - INFO - connected 2023-01-11 01:57:42 [237ms] [Error] [omni.core] [18882] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [18880] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:57:42 [263ms] [Info] [carb.launcher.plugin] [18881] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:42 [263ms] [Info] [carb.launcher.plugin] [18881] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 18880, bytes = 0} 2023-01-11 01:57:42 [263ms] [Info] [carb.launcher.plugin] [18881] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 18880. 2023-01-11 01:57:42 [263ms] [Info] [carb.launcher.plugin] [18882] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:42 [263ms] [Info] [carb.launcher.plugin] [18882] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 18880, bytes = 0} 2023-01-11 01:57:42 [263ms] [Info] [carb.launcher.plugin] [18882] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 18880. 2023-01-11 01:57:44 [2,004ms] [Info] [omni.core] [18871] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:57:44 [2,004ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:57:44 [2,004ms] [Info] [omni.core] [18871] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:57:44 [2,004ms] [Info] [omni.core] [18867] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:57:44 [2,004ms] [Info] [carb] [18867] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:57:44 [2,005ms] [Warning] [omni.keyvaluedatabase.leveldb] [18867] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:57:44 [2,005ms] [Info] [omni.core] [18871] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:57:44 [2,029ms] [Warning] [omni.keyvaluedatabase.leveldb] [18867] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:57:44 [2,029ms] [Fatal] [omni.core] [18867] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta' 2023-01-11 01:57:50 [0ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:57:50 [0ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:57:50 [0ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:50 [0ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:50 [0ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:50 [0ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:57:50 [0ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:50 [0ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:57:50 [0ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:50 [1ms] [Info] [carb] [19084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:50 [1ms] [Info] [omni.server] [19084] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:57:50 [1ms] [Info] [omni.server] [19084] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:57:50 [1ms] [Info] [omni.server] [19084] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:57:50 [2ms] [Info] [omni.server] [19084] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:57:50 [2ms] [Info] [omni.server] [19084] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:57:50 [2ms] [Info] [omni.server] [19084] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:57:50 [2ms] [Info] [omni.server] [19084] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:57:50 [2ms] [Info] [omni.server] [19084] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:57:50 [2ms] [Info] [omni.server] [19084] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:57:50 [2ms] [Info] [omni.server] [19084] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:57:50 [2ms] [Info] [carb] [19084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:50 [2ms] [Info] [carb] [19084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:50 [2ms] [Info] [carb] [19084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:50 [3ms] [Info] [carb] [19084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:50 [3ms] [Info] [carb.launcher.plugin] [19084] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1e72ad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:57:50 [3ms] [Info] [carb.launcher.plugin] [19084] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 19084. 2023-01-11 01:57:50 [3ms] [Info] [carb] [19084] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:57:50 [3ms] [Info] [carb] [19084] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:57:50 [4ms] [Info] [carb.launcher.plugin] [19096] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 19097 from the parent 19084. 2023-01-11 01:57:50 [4ms] [Info] [carb.launcher.plugin] [19096] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 19097. 2023-01-11 01:57:50 [5ms] [Info] [carb.launcher.plugin] [19096] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 19097. 2023-01-11 01:57:50 [5ms] [Info] [carb.launcher.plugin] [19084] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 19097 from the parent 19084. 2023-01-11 01:57:50 [5ms] [Info] [carb.launcher.plugin] [19084] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 19097 {process = 0x1e72ad0} 2023-01-11 01:57:50 [5ms] [Info] [omni.core] [19084] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:57:50 [5ms] [Info] [carb.launcher.plugin] [19098] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 19097. 2023-01-11 01:57:50 [5ms] [Info] [omni.core] [19084] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:57:50 [5ms] [Info] [carb.launcher.plugin] [19099] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 19097. 2023-01-11 01:57:50 [224ms] [Error] [omni.core] [19099] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:57:50 [227ms] [Error] [omni.core] [19099] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:50,266 - system_log - INFO - reconnecting ... 2023-01-11 01:57:50 [235ms] [Error] [omni.core] [19099] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:50,274 - system_log - INFO - connected 2023-01-11 01:57:50 [236ms] [Error] [omni.core] [19099] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use [19097] 2023-01-11 01:57:50 [236ms] [Error] [omni.core] [19099] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:57:50 [263ms] [Info] [carb.launcher.plugin] [19099] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:50 [263ms] [Info] [carb.launcher.plugin] [19099] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 19097, bytes = 0} 2023-01-11 01:57:50 [263ms] [Info] [carb.launcher.plugin] [19098] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:50 [263ms] [Info] [carb.launcher.plugin] [19099] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 19097. 2023-01-11 01:57:50 [263ms] [Info] [carb.launcher.plugin] [19098] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 19097, bytes = 0} 2023-01-11 01:57:50 [263ms] [Info] [carb.launcher.plugin] [19098] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 19097. 2023-01-11 01:57:51 [0ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v2.0]) (impl: omni.meta-upgrade) 2023-01-11 01:57:51 [0ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:57:51 [0ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:51 [0ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:51 [0ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:51 [0ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.verifier (interfaces: [omni::server::IVerifier v1.1]) (impl: omni.verifier) 2023-01-11 01:57:51 [0ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:51 [0ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects) 2023-01-11 01:57:51 [0ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:51 [0ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():391: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server) 2023-01-11 01:57:51 [1ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v2.2],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin) 2023-01-11 01:57:51 [1ms] [Info] [omni.server] [19148] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():151: Launching C++ Omniverse API Server version (1.0.84b6cd59) API version(1.20) 2023-01-11 01:57:51 [1ms] [Info] [omni.server] [19148] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():153: Build from commit 84b6cd59 on Tue Dec 6 07:57:56 2022 -0800 2023-01-11 01:57:51 [1ms] [Info] [omni.server] [19148] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():98: Starting up Omniverse Server. 2023-01-11 01:57:51 [1ms] [Info] [omni.server] [19148] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():71: ============================================================================== 2023-01-11 01:57:51 [1ms] [Info] [omni.server] [19148] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():72: Default Data Directory: /home/nw/.local/share/ov/data/server 2023-01-11 01:57:51 [1ms] [Info] [omni.server] [19148] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():73: Default Log Directory : /home/nw/.cache/ov/server 2023-01-11 01:57:51 [1ms] [Info] [omni.server] [19148] ../../../source/plugins/omni.server/Globals.cpp:initDefaultDirs():74: ============================================================================== 2023-01-11 01:57:51 [1ms] [Info] [omni.server] [19148] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():611: Setting Log directory to: /home/nw/.nvidia-omniverse/logs/Nucleus 2023-01-11 01:57:51 [1ms] [Info] [omni.server] [19148] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings: 2023-01-11 01:57:51 [2ms] [Info] [omni.server] [19148] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: { "LFT": { "INTERFACE": "127.0.0.1" }, "log": { "fileLogLevel": "verbose", "detail": false, "outputStream": "stderr", "includeThreadId": true, "includeLineNumber": true, "level": "Info", "outputStreamLevel": "verbose", "enableDebugConsoleOutput": true, "includeTimeStamp": true, "debugConsoleLevel": "verbose", "includeFunctionName": true, "sources": { "omni.server.DELTA_CONTENT": { "enabled": false }, "omni.server.DBCONTENT": { "enabled": false }, "omni.server.CONNECTIONS": { "enabled": false }, "omni.server.OMNI_OBJECTS": { "enabled": false }, "omni.server.EVENT_SYSTEM": { "enabled": false }, "omni.server.API_SERVER": { "enabled": false }, "omni.server.API_CLIENT": { "enabled": false }, "omni.server.FRAMING": { "enabled": false }, "omni.server.SHUTDOWN": { "enabled": false } }, "file": "/home/nw/.nvidia-omniverse/logs/Nucleus/omni.server.app.log", "includeFilename": true, "fileFlushLevel": "verbose", "setElapsedTimeUnits": "milliseconds", "enableColorOutput": true, "enableStandardStreamOutput": true, "enabled": true, "async": false, "fullDetail": false, "fileAppend": true, "flushStandardStreamOutput": false, "processGroupId": 0, "includeSource": true, "includeChannel": true, "includeProcessId": false }, "pluginSearchPaths": [ "." ], "LOG": { "DIR": "/home/nw/.nvidia-omniverse/logs/Nucleus" }, "omni.common.settings": { "env.var.prefix": "OMNI_" }, "APILOGGING": 1, "crashreporter": { "compressDumpFiles": false, "overrideDefaultDumpFlags": false, "url": "https://services.nvidia.com/submit", "log": "true", "product": "OmniverseNucleus", "uploadTimeoutMs": 7200000, "enabled": "true", "preserveDump": false, "skipOldDumpUpload": false, "version": "114.4+master.gitlab.21980.84b6cd59", "retryCount": 10, "data": { "buildBranch": "master", "buildNumber": "21980", "buildMr": "0", "buildHash": "84b6cd59", "buildVersion": "114.4.0", "buildMinor": "4", "buildMajor": "114", "buildPatch": "0", "buildCi": "gitlab" }, "alwaysUpload": false, "dumpFlags": "", "debuggerAttachTimeoutMs": 0, "dumpDir": "." }, "NUCLEUS": { "SERVER": { "INTERFACE": "127.0.0.1" } }, "S3": { "RESOLVER": { "CACHE": "localhost:8891" } }, "plugins": { "omni.metrics-prometheus": { "predefined_labels": { "omni_instance": "" }, "model": "pull", "push": { "host": "", "port": 9091, "flushTimeInSeconds": 5 }, "metricsPrefix": "omni_server_api", "pull": { "host": "127.0.0.1", "endpoint": "/", "port": 3010 } }, "carb.tasking.plugin": { "debugLocks": false, "debugTaskBacktrace": false, "threadCount": "4", "fiberCount": 4095, "stuckCheckSeconds": 1, "resetFiberStack": true, "debugWaitingTasks": false }, "omni.meta-upgrade": { "interactive": false, "exit_on_any_failure": false, "enabled": "true", "repair": { "enabled": "true", "only_after_upgrade": "true" }, "report_progress_every": 1000, "convertors": { "meta": { "dry_run_since_version": -1 } }, "remove_deduplicated_content": true, "content_resolver": { "resolver": "omniverse_resolver_file", "options": { "async": true, "root": "" } } }, "omni.resolver": { "directory": ".", "libraries": [ "omniverse_resolver_file", "omniverse_resolver_memory", "omniverse_resolver_composite", "omniverse_resolver_leveldb", "omniverse_resolver_s3" ] }, "omni.delta-applier": { "ports": { "server_port": 3001 } }, "omni.server": { "store_crash_repro": true, "omniobjects.cache": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "api_acls": { "unlock": "w", "lock": "w", "rename": "a", "set_path_options": "w", "set_acl_v2": "a", "get_acl_v2": "r", "create_directory": "w", "unmount": "w", "checkpoint_version": "w", "delete": "a", "create_object": "w", "list": "r", "copy": "w", "mount": "w", "create_asset": "w", "read": "r", "get_acl_resolved": "r", "get_checkpoints": "r", "update": "w" }, "usergroups": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "usergroups.1.0", "delete_on_startup": "false" } }, "metadata": { "plugin": "omni.keyvaluedatabase.leveldb", "options": { "async": true, "root": "", "subdir": "meta.1.1", "delete_on_startup": "false" } }, "run": { "versioning": "true" }, "endpoints": { "api": [ { "interface": "127.0.0.1", "lft_address": "http://*:3030", "port": "3009", "secure": false } ], "lft": [ { "interface": "127.0.0.1", "port": "3007", "secure": false } ], "service": [ { "interface": "127.0.0.1", "port": "3006", "secure": false } ] }, "dirs": { "omniobjects_dir": "/home/nw/.local/share/ov/data/server", "usergroups_dir": "/home/nw/.local/share/ov/data/server", "log_dir": "/home/nw/.nvidia-omniverse/logs/Nucleus", "metric_dir": "/home/nw/.local/share/ov/data/server", "content_dir": "/home/nw/.local/share/ov/data/server", "record_dir": "", "meta_dir": "/home/nw/.local/share/ov/data/server" }, "home_folder_default_acls": { "owner": "rwa", "rest": { "users": "", "gm": "rwa" } }, "master_password": "omniverse", "discovery": { "port_based_default_port": 3333, "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }", "enabled": "true", "token": "test", "host": "localhost" }, "admin_folders": [ "/Users/omniverse/", "/Users/deeptag_service/", "/Users/search_service/", "/Users/tags_service/", "/Users/thumbnails_service/", "/Users/index/", "/.system/" ], "test_token_prefix": "", "omniobjects.values": { "resolver_kind": "omniverse_resolver_file", "logging_enabled": "true", "resolver_options": { "async": true, "root": "" } }, "file_and_username_filter_exclude_regex": "", "content_resolver": { "delete_on_startup": "false", "resolver": "omniverse_resolver_composite", "options": { "total_size_high": 6442450944, "total_size_low": 4294967296, "number_elems_high": 120000, "persistent_options": { "async": true, "root": "" }, "persistent_kind": "omniverse_resolver_file", "temporal_options": {}, "number_elems_low": 100000, "elem_size_max": 1048576, "temporal_kind": "omniverse_resolver_memory" }, "logging_enabled": "true" }, "delta_content_bw_estimate": 0.5, "content_flush_time_sec": 10, "default_s3_proxy_cache": "", "disable_passwords": false, "content_flush_enabled": true, "peer_messaging": { "channel": "omni_server_messages", "enable": "false", "application_name": "Nucleus API", "exit_on_connection_failure": "false", "payloads": { "create_event": { "name": "create" }, "list_event": { "name": "list" }, "servernotification_event": { "name": "servernotification" } } }, "metrics": { "enabled": "false", "request": { "duration": { "buckets": [ 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000.0, 50000.0, 100000.0, 1000000.0 ] } } }, "content_type": "", "auto_create_users": true, "default_acls": { "owner": "rwa", "rest": { "gm": "rwa" } }, "max_full_path_length": "16384", "concurrent_mount_commands": true, "quotas": { "list_subscriptions": "0", "read_subscriptions": "0" }, "password_recovery": true, "non_utf8_paths_allowed": "false", "notifications": {}, "file_system_table": true, "delta_content_hang_timeout": 2.0, "api_logging": { "json_outputs": {}, "enable_legacy_system": "1", "source_base_name": "api-server", "event_buffer_min_size_bytes": 1024, "tsv_outputs": [ { "omit_unknown_columns": true, "print_header_on_new_field": true, "column_order": [ "source", "time", "id", "type", "requestname", "status", "reqid", "username", "connectionid", "clientip", "uri", "eventsize", "optionalinfo", "branch", "checkpoint", "multiplepath" ], "files": [ "eventlog.tsv" ], "omit_columns": {} } ], "source_instance_name": "", "event_buffer_max_size_bytes": 1048576, "file_settings": [ { "filename": "eventlog.tsv", "flush_frequency": 1 } ], "event_buffer_try_shrink_interval_sec": 10, "enable": "false", "omit_event_types": {} }, "lft": { "launch": "true", "check_alive_interval": 10, "threshold": "65536" }, "allowed_users": {}, "default_folders": [ [ "/", { "users": "r", "gm": "rwa" } ], [ "/Users/", { "users": "r", "gm": "rwa" } ], [ "/Projects/", { "users": "rw", "gm": "rwa" } ], [ "/Library/", { "users": "rw", "gm": "rwa" } ] ], "crash_record_recovery": true, "mount_management_requires_admin": true, "auth_service_client": { "enabled": true, "deployment_lookup": "internal", "override_service_address": "" }, "removed_content_pending_time_sec": 5, "delta_manager": { "ignorelist_retry_delay_ms": 5000, "num_retries_until_ignored": 10, "delta_apply_limit_mb": 128, "ping_timeout_ms": 1000, "delta_apply_throttle_ms": 500, "delta_apply_timeout_ms": 5000, "reconnect_timeout_s": 20, "ping_interval_on_startup_ms": 100, "bw_estimate_bps": 256, "ping_interval_ms": 1000 }, "metric_period": 10.0, "token_expire_time": 86400, "max_path_piece_length": "255", "record_integrity_check": false, "metacache_flush_count": 1000, "delta_content_ping_interval": 0.0, "list_batch_size_limit": 100, "default_branch_name": "default", "mounts": { "default_mounts": [ { "path": "/NVIDIA", "options": "{\"timeout\": 600, \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}", "resolver": "omniverse_resolver_s3", "override": true } ], "logging_enabled": "false", "allowed_mount_resolvers": [ "omniverse_resolver_file", "omniverse_resolver_s3" ], "allowed_file_resolver_roots": {} }, "auth": { "connection_id_salt": "7b6a5f4e0a1b2c3d", "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----" }, "lft_content_expiration_time": "600", "allowed_users_legacy": {}, "omniobjects": { "content_mem_limit": 268435456, "mem_vs_file_threshold": 262144, "max_objects_in_cache": 1024, "options": { "async": true, "root": "", "subdir": "omniobjects.1.0", "delete_on_startup": "false" }, "max_content_file_cache_size": 1048576000, "max_content_mem_cache_size": 104857600, "unpack_object_buffer_size": 33554432, "plugin": "omni.keyvaluedatabase.leveldb", "lft_read_timeout_seconds": 3600, "lft_multi_content_threshold": 0, "ext_value_min_size": 262144 }, "multipart_chunk_size_mb": "5" }, "omni.websocketslib": { "tracing": false, "defaults": { "tcp_keepalive_interval": 2, "tcp_keepalive_probes": 2, "tcp_keepalive_time": 10, "http_max_header_length": "61440", "tcp_timeout_seconds": 10 } }, "omni.messaging-redis": { "password": "yBGfkZ0iRT", "reconnect_interval_ms": 1000, "port": "6379", "timeout_ms": 500, "host": "kube-msk-redis.nvidia.com", "max_reconnects": 5 } }, "pluginsLoaded": [ "carb.dictionary*.plugin", "carb.settings.plugin", "carb.launcher.plugin", "carb.tasking.plugin", "carb.crashreporter-breakpad.plugin", "omni.metrics-null", "omni.signalhandler", "omni.wakeupthread", "omni.eventsystem", "omni.keyvaluedatabase.leveldb", "omni.keyvaluedatabase.redis", "omni.server", "omni.websocketslib", "omni.resolver", "omni.core", "carb.omnitrace.plugin", "omni.objects", "omni.meta-upgrade", "omni.verifier", "omni.messaging-redis", "omni.eventlogger" ], "omni.server.app": {}, "privacy": { "performance": "false" } } 2023-01-11 01:57:51 [2ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib) 2023-01-11 01:57:51 [2ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.metrics-null (interfaces: [omni::server::IMetrics v1.1]) (impl: omni.metrics-null) 2023-01-11 01:57:51 [2ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core) 2023-01-11 01:57:51 [2ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.3]) (impl: carb.launcher.plugin) 2023-01-11 01:57:51 [2ms] [Info] [carb.launcher.plugin] [19148] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():230: created the process object 0x1dccad0 for './omni-lft-service/omni-lft-service'. 2023-01-11 01:57:51 [2ms] [Info] [carb.launcher.plugin] [19148] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():643: launching the process './omni-lft-service/omni-lft-service' from process 19148. 2023-01-11 01:57:51 [2ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin) 2023-01-11 01:57:51 [2ms] [Info] [carb] [19148] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread. 2023-01-11 01:57:51 [3ms] [Info] [carb.launcher.plugin] [19171] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():883: [child]: starting the child process 19172 from the parent 19148. 2023-01-11 01:57:51 [3ms] [Info] [carb.launcher.plugin] [19171] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():892: [child]: setting the parent death signal for the child process 19172. 2023-01-11 01:57:51 [4ms] [Info] [carb.launcher.plugin] [19171] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():1061: [parent]: successfully launched the child process 19172. 2023-01-11 01:57:51 [4ms] [Info] [carb.launcher.plugin] [19148] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():775: [parent]: launched the child process 19172 from the parent 19148. 2023-01-11 01:57:51 [4ms] [Info] [carb.launcher.plugin] [19148] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():798: [parent]: successfully launched the child process 19172 {process = 0x1dccad0} 2023-01-11 01:57:51 [4ms] [Info] [omni.core] [19148] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/home/nw/.local/share/ov/pkg/nucleus-workstation-2022.4.1/Nucleus/./omni-lft-service/omni-lft-service) 2023-01-11 01:57:51 [4ms] [Info] [omni.core] [19148] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up 2023-01-11 01:57:51 [4ms] [Info] [carb.launcher.plugin] [19174] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stderr reader' for the child process 19172. 2023-01-11 01:57:51 [4ms] [Info] [carb.launcher.plugin] [19173] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():546: starting the read thread for 'launcher stdout reader' for the child process 19172. 2023-01-11 01:57:52 [248ms] [Error] [omni.core] [19174] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /home/nw/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers 2023-01-11 01:57:52 [252ms] [Error] [omni.core] [19174] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:52,070 - system_log - INFO - reconnecting ... 2023-01-11 01:57:52 [262ms] [Error] [omni.core] [19174] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2023-01-11 10:57:52,080 - system_log - INFO - connected 2023-01-11 01:57:52 [262ms] [Error] [omni.core] [19174] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last): File "run_server.py", line 3, in File "lft/server.py", line 801, in run_application File "asyncio/base_events.py", line 616, in run_until_complete File "lft/server.py", line 715, in startup_server File "aiohttp/web_runner.py", line 121, in start File "asyncio/base_events.py", line 1463, in create_server OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 3030): address already in use 2023-01-11 01:57:52 [262ms] [Error] [omni.core] [19174] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:[19172] Failed to execute script 'run_server' due to unhandled exception! 2023-01-11 01:57:52 [294ms] [Info] [carb.launcher.plugin] [19174] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:52 [294ms] [Info] [carb.launcher.plugin] [19173] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:read():234: read ended {errno = 0/Success} 2023-01-11 01:57:52 [295ms] [Info] [carb.launcher.plugin] [19174] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 19172, bytes = 0} 2023-01-11 01:57:52 [295ms] [Info] [carb.launcher.plugin] [19174] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stderr reader' for the process 19172. 2023-01-11 01:57:52 [295ms] [Info] [carb.launcher.plugin] [19173] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():596: read failed. Exiting the read thread {pid = 19172, bytes = 0} 2023-01-11 01:57:52 [295ms] [Info] [carb.launcher.plugin] [19173] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():626: done the read thread for 'launcher stdout reader' for the process 19172. 2023-01-11 01:57:53 [2,004ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v7.0]) (impl: omni.resolver) 2023-01-11 01:57:53 [2,004ms] [Info] [omni.core] [19156] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():308: discovery client: starting up 2023-01-11 01:57:53 [2,004ms] [Info] [omni.core] [19156] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():351: discovery client: connecting to ws://localhost:3333 2023-01-11 01:57:53 [2,004ms] [Info] [omni.core] [19148] ../../../source/plugins/omni.core/impl/CoreImpl.cpp:getOmniResolverInstance():87: Successfully created OmniResolver instance of type(omniverse_resolver_composite) 2023-01-11 01:57:53 [2,004ms] [Info] [carb] [19148] ../../../source/framework/PluginImpl.cpp:initialize():512: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb) 2023-01-11 01:57:53 [2,004ms] [Warning] [omni.keyvaluedatabase.leveldb] [19148] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():74: wasn't able to load the meta database, trying to repair it ... 2023-01-11 01:57:53 [2,005ms] [Info] [omni.core] [19156] ../../../source/plugins/omni.core/impl/ServicesClients.cpp:run():386: discovery client: registering 'external' with transport 'connlib' and transport parameters '{ "url": "ws://127.0.0.1:3009" }' 2023-01-11 01:57:53 [2,029ms] [Warning] [omni.keyvaluedatabase.leveldb] [19148] ../../../source/plugins/omni.keyvaluedatabase.leveldb/leveldb.cpp:LevelDBImpl():79: repair failed 2023-01-11 01:57:53 [2,029ms] [Fatal] [omni.core] [19148] ../../../source/plugins/common/KvDbAsync.h:Instance():36: error while initializing database 'meta'