Security inside job

AI & Data Science Cybersecurity
,
1

Hello community,

I’m the founder of a startup in the Inception program, focused on developing a green hosting platform. Our project has been stalled by a complex security incident, and I’m hoping to get some input since we can’t exactly afford a cybersecurity firm.

This isn’t a standard support request, but rather a complex technical puzzle involving a popular hosting software vendor.

Technical Timeline & Key Findings:

The incident unfolded in several distinct phases:

  • Phase 1: Initial Instability: The incident began with repeatable SIGSEGV (Segmentation Fault) crashes on a new dedicated server, traced directly to the vendor’s core php-fpm process by apport logs. This instability was reported but dismissed as a non-security issue by the vendor’s support management.

  • Phase 2: Architectural Anomalies: A deeper investigation of the production environment revealed an undeclared stack running on my server, including GraphQL and Sentry, with all traffic being proxied through Caddy. This reverse proxy was installed by a “security” module from their marketplace. Support claimed this component was non-operational, yet it remained active and running on the server (visible via SSH) even after the module was uninstalled and the subscription had lapsed.

  • Phase 3: Credential Compromise via Supply Chain: My entire password vault was compromised. The entry vector was the vendor’s own authorized backup application, which had privileged OAuth access to my OneDrive and accessed a forgotten password export file stored there.

  • Phase 4: Escalation & Subsequent Events: Immediately following my escalation of these findings to the vendor’s DPO, a series of hostile events began:

    • A persistent, high-volume Layer 7 botnet assault was launched against my new infrastructure from a diverse range of cloud providers (Hetzner, Alibaba, etc.).

    • I found, by Cloudflare, an email in my name sent to aws, which it wasn’t me.

    • I’m being monitored on every single website i put online by this software even when i don’t use it anymore, as i stopped 04/08, among other stuff.

Evidence Available for Private Review: I have a comprehensive dossier with logs, screenshots, and documentation for every claim made above, including a Cloudflare audit log showing the third-party security module creating firewall rules via API key.

My Questions for this Community:

  1. From a purely technical standpoint, does the combination of an undeclared reverse proxy (Caddy), GraphQL, and Sentry on a customer server constitute a potential architectural backdoor for traffic inspection or data exfiltration?

  2. What are the recommended forensic next steps for analyzing a persistent, “zombie” component like Caddy to determine its activity after the parent application has been removed?

  3. For those who have dealt with complex security incidents involving software vendors, what are the best practices for disclosure and interaction when direct communication appears to trigger escalating hostile activity?

I’m looking for guidance or any kind of help is welcomed, thank you.

2

Hello Cesare,

Before assessing any of this I believe we are missing some information of the underlying infrastructure, is this in the cloud? if so which one? SaaS/PaaS? I am assuming there is some sort of high-priviledge add-on in your environment from a potentially not-trusted or compromised vendor, that was leveraged to compromise your account.

Anyhow, after some assumptions - addresing your questions:

  1. Yes, a reverse proxy that can intercept queries is usually a good sign of a backdoor.
  2. Before removing the “zombie” agent, I would highly recommend properly spending some time scoping the incidents, what we see more often than not with companies rushing to “kick the attackers out” as fast as possible, is that they do not fix where they came from, so they just come back again. Said that, figure out where the persistance mechanism is, whether on cloud, on prem. on an identity, cron job, etc.. Some basic guidance on this without knowing the specifics would be to check processes, grep for caddy, check the conf file, and wireshark or tcpdump for any abnormal communications to the outside. File forensics if you suspect there is an executable somewhere, or memory analysis if its a living of the land or memory-only malware/implant.
  3. Depends on regulators, your location, if you have contractual reporting obligations to clients, police, etc.. missing some context here.

Below some generic guide/help to address the incident:

Immediate Actions:

  1. Isolate affected systems

  2. Preserve evidence - Create forensic images before cleanup (for analysis)

  3. Change ALL credentials - Assume complete compromise (if you do not know the scope, assume the worst)

  4. Document everything with timestamps

Legal/Disclosure Strategy:

  1. Engage law enforcement - This appears to involve criminal activity (botnet type of activite, farming maybe?)

  2. Contact relevant data protection authorities (depending on your jurisdiction)

  3. Consider coordinated disclosure through a third party rather than direct communication

Evidence Preservation:

  • Keep multiple copies of your evidence in different locations

  • Consider using a secure evidence sharing platform

  • Document the chain of custody

Hope that helps

Best,

Diego