The saga began when my XMPP server was malfunctioning and I couldn't debug it properly (or just couldn't be bothered), so I simply reinstalled it. But I forgot to backup account files of some people that were using it, so I tried photorec to recover. I was then quite surprised how my (virtual) drive filled up in seconds. I instantly realized what was up... Curiosity got her way and I downloaded everything that had been lifted and started checking it out, and was shocked. Shocked to find images with doxing info (names), for example. Someone more evil than me might have abused it right then... I couldn't even be bothered to look through it all but I'm sure I'd be able to find more spicy stuff there if I cared to.
Over 6 thousand PNGs lifted in a few seconds by photorec
If you still didn't figure out what this is about, photorec somehow bypasses Incognet's virtualization. I have access to many files that I surely did not put there. Executables, sqlite databases (found many on my first try, now only one seems to be recoverable), and who knows what else that a determined attacker might explore for clues. Let me show you an example image I've found:
I don't even know what IPtraf is, nor have I ever set up anything that takes screenshots, yet I ended up lifting about 20 similar images (among other ones). What does it mean? Don't upload anything to Incognet that you don't want others to see (or encrypt it by GPG, zip password, etc...). The link being secret won't save you because photorec goes after the underlying data. Your nudes, your medical information, your anything might be available to anyone that dares to try file recovery at some point. And certain things, like the databases, might be impossible to encrypt so this vuln seems not even completely mitigatable in principle (on the user's side anyway). I found this out by total accident so it's surely abusable by even amateurs. But no one has written about it yet, as far as I can see.
Why am I doing so, though? I reported the issue first by mail, and got ignored. So I was forced to use their slow, annoying and unreliable portal instead, and got quite angry, which sadly reflected in the ticket I've posted. Keep in mind that at this point there has been zero indication that they don't reply to E-mails (and no other host I've deal with ever worked that way...). I was hoping that I'd be able to use E-mail for subsequent communication, at least, but Incognet seems to be ignoring all mail. In fact, after I called them out about it, they began starting every message sent with DO NOT RESPOND TO THIS EMAIL, NO ONE WILL SEE IT!
which leaves no doubt. They really want you to jump through their insane hoops (the portal logs you out constantly) to report even such a critical issue. And they still haven't done anything about it (I just confirmed it now; and it's been 3 weeks!). So your nudes and other things might be available for others to see right now. And again, this doesn't require super special hacking skills to use, so I'd rather tell my readers to beware of the stuff you upload there than hide the vuln and hope that (malicious) people won't figure it out on their own regardless. By the way, Incognet confirmed the vuln to me in one of their E-mails:
It's strange though, because in my original test I did download a large random icon image pack, that had a bunch of random .png web icons for web-design. I extracted it, but after reinstalling the OS and running photorec, these items were not discovered. Only the random OS documentation junk as described above.
Icons for web design - does it look like something that an OS has by default? After this E-mail, though, they tried to gaslight me into believing that that's actually the case (no, Slackware doesn't have thousands of PNGs, nor a bunch of sqlite databases...), or that "their virtualizor spits junk" (if so, it's a weirdly specific kind of junk, and there is quite a lot of it!). They even tried to threaten me into updating my blog
once I get a better understanding
. How patronizing, and that right after they admitted they got results they couldn't explain from my exact test!
The Incognet saga continued when my domain was about to expire in April (of 2025). On March 20, they graciously sent me an E-mail telling me they'll kill it if I don't renew it by then. That's not really the problem, they are a business after all. The problem was that their portal wasn't working for weeks already at the time of the E-mail. In the E-mail, they also said to please reply to this email
if I have any questions
, but when I asked How can I renew when your portal doesn't load?
I got completely ignored. So I ended up with literally no way to renew it. And neither their knowledgebase nor their announcements could be reached at all (I had this confirmed by a MUC member so it wasn't only on my end). That is, until another MUC member notified me that their I2P mirror (that I had no idea even existed) worked. So I was able to log in and see that I did in fact have auto-renew enabled, and monero already in there - but imagine I didn't? I would have lost my domain without recourse - and surely some people did suffer that fate. I wanted to leave Incognet completely after this fiasco, but I could not find a way to transfer my domain. They say it exists and it probably does, it's just not obvious enough, and their knowledgebase does not provide the exact steps. Edit: I searched through everything, and there just isn't a way to transfer domains, unless you have to perform a rain dance to make the option show up. One person who used Incognet before told me she did transfer "easily", which suggests either a conspiracy agaist poor diggy, or some technical issue (that they haven't bothered to fix in literal years). Also posted a support ticket trying to get them to give me the domain transfer code, or just fix whatever's preventing the display; got ignored for 8 or so days (five business days is supposed to be the limit of getting to a support ticket), CBF counting anymore. In fact, I held back on this report purely in hopes of recovering my domain, but at some point the likelihood fell below believable threshhold, I faced the looming danger of losing it, and finally pulled the trigger. Surely they will be eager to give it back to me after reading this :D.
Part 3 of the Incognet saga began when - on April 16 - they sent me (and everyone else who had a Netherlands VPS) an E-mail mentioning plans of relocating operations
to a new facility in Amsterdam
because of rising costs
in the old location. They warned that in early May there will be service-impacting maintenance
, and that they will send additional notifications once the dates come closer. On June 19 (not quite early May, but whatever...), they did send another E-mail, saying that your VPS will be migrated in the coming week
. So far so good. I got told The service disruption may be as short as a couple of hours or as long as 24 hours
. Fine, that much downtime I can take - I thought. On June 25, they sent the final E-mail, pinning the relocation date at June 27, and saying the migration will take many hours, possibly more than a day
. Again, I was fine with this duration, so I did no preparations, and just decided to do some writing I've been putting off, and wait. Up until this point, everything was acceptable to me.
Yet the days kept passing while my server kept not working. Hey, I thought it was supposed to be a few hours long painting job, not an entire home renovation. I decided to visit Incognet's announcement page (archive) (MozArchive) to figure out what's going on. A post on June 29th, 3:00PM EST, said:
Migrations are now 100% complete. All services that were previously in Naaldwijk, Netherlands are now in Amsterdam, Netherlands. For the next 24 hours there will be network quirks as a result of our IP subnets being announced in a new location while some BGP routes beyond our control may still direct traffic to the old location. This will resolve itself. To address a concern related to packetloss, there may be a planned maintenance in the coming week to replace a piece of hardware that may be the cause. If so, we will announce this maintenance with as much notice as possible.
Yet when I attempted to turn on my VPS (I expected it to do that by itself, TBH) from the portal, I saw this:
The first time I tried was on June 29, the first day the migrations were supposed to be finished. Then the next day, then the next, and the next (today, July 2) - with no effect aside from some burned neurons while their retarded portal kept logging me out every few minutes. I truly was hoping Incognet would end up not being complete failures just once, with something so critically important on the line. But as usual, they could not do anything properly, and I once again ended up trapping myself inside my own cage of naivety. Of course, an idea came to mind that this is just a conspiracy to get rid of me, after the vuln episode - but it's also possible they are simply that incompetent. UPDATE: the situation still persisted on July 18, when I've deleted the server. Who knows how long it would have continued.
I waited until July 3 in hopes they would graciously recover my VPS but they did not. Even though I really didn't want to do it, I had to succumb to the last "weapon" available to me, which was inviting a friend (as I really didn't want to talk to them directly) to post a support ticket in their retarded portal that logs you out every few minutes (which he did also end up complaining about). He posted this:
Hi,
The mentioned VPS status is "offline" despite the blog statement that migration was completed (See https://portal.incognet.io/serverstatus.php?view=resolved for reference) I'm getting this error "1) VPS is locked, hence no actions are allowed to perform - VPS migration is in process"
Please review and make sure it works.
Thanks
After this, they sent an automated E-mail which said (among other things):
Please allow 1-5 business days for most requests. All tickets are responded to in the order that they're received and your patience is appreciated.
Of course, on day 6 there was still no reply. This appears to be common in that one of my friends has a several months old (high priority) rotting ticket that is still unresolved, and I myself have one from 2023 which requested a pretty important feature that has so far not been implemented (and yet would be really easy to). Continuing:
Practically all products are considered "self managed", we do not advertise any support level guarantees.
Translation: "we can and will break your shit without recourse" (by the way, there is a significant number of VPSes who do actually take responsibility for breaking shit, codified right into their ToSes, so succumbing to this bullshit isn't necessary). Now put away your coffee if you are holding one:
We have extensive internal monitoring that gives us immediate alerts of events that would cause service level disruption to customers. These alerts are acted on and reviewed immediately, 24/7.
Was the system not extensive
enough to notify them about completely nonfunctioning servers? Of course (like I said earlier), another option is that this is an attempt to get rid of me with some plausible deniability - and it's looking more and more likely with each passing fuckup. The alternative - that the super duper advanced system is a mirage - might actually be even worse for them, though. It would mean everyone is in danger of having their server go inexplicably down - not just the spicy.
UPDATE: HAHA, a new announcement (archive) (MozArchive) just dropped (12 days after migrations were supposed to be finished, and everything back to normal), and it is just as shameless and pathetic as you'd imagine:
Many customers have reported network stability issues, high packetloss, inconsistent or slow network speeds as well as unexpected/unannounced VM reboots after the migration from Naaldwijk to Amsterdam.
What about completely nonfunctioning servers? Why not admit what's actually going on instead of hiding behind vague terms, and pretending the services are still working, just with some hiccups?
Unfortunately, after the migration began, network related issues occurred that were not present in the initial testing/benchmarking phase of the new location. Only after live-traffic was present on the network did these issues being to appear, which has led to the issues you have experienced.
Shouldn't the super advanced automated monitoring system have reported those?
Friday, July 11th, there is planned maintenance to correct these issues. Scheduled to arrive at the datacenter in Amsterdam tomorrow is new hardware which will be used to replace what we believe is some faulty components.
Can anyone in there even insert a hard disk?
There will be downtime during this maintenance, but we do believe that this will resolve the issue moving forward.
There is already downtime of 14 days, so nothing changes?
We understand the frustration of all of those impacted by this, and we are quite frustrated by the situation ourselves. Our Netherlands based services have always been top notch and of high quality, and after the migration the quality fell well below not only our standard, but the standard of any serious business. We cannot apologize enough for the inconvenience caused by this and are working diligently on restoring service to a production quality state.
No, you don't care about anyone's frustration, because you are ignoring tickets reporting completely nonfunctioning servers. A production quality state
? You don't even have a state at all, because the thing that's supposed to have a state isn't turning on.
This entire situation is truly insane. I mean, I can understand hardware failures, migrations, network problems, whatever. I can even accept taking a little more time to fix the issues than seems needed. What I cannot tolerate is the absolute disregard for your customers by pretending stuff is fixed when it's not, minimizing the scope of the damage, gaslighting, ignoring support tickets while lying about the time you take to answer them, imagining "advanced monitoring systems" that conveniently miss critical issues, and all this while continuing to steal the money for a totally nonfunctional service.
UPDATE 2: they started blaming (archive) (MozArchive) their issues on broken computer parts. So they ordered more (why not have them at hand?) and they got stuck in Italy. The (clown)show must go on! All while servers like mine still refuse to turn on. Of course, 3 days after they (supposedly - who knows if all of this isn't simply made up?) put in the parts, stuff still didn't work, so they had to order even more (archive) (MozArchive) - We have on order additional components that we expect to arrive by the week's end
. What's going to happen now? A plane will mysteriously travel to the Pacific Islands where the dwellers will rob it? What prevents them from continuing to use this excuse indefinitely? Why does it take so long to replace some parts in the first place? Why is more than one try needed? Why can't they still admit to having completely nonfuctional servers?
Because of Incognet's fuckups, I had to recreate my setup on another host (that I fortunately already had), then redirect the domain there. What if I didn't have the additional host? What if I was too poor to buy one? What if domains also stopped working (since Incognet was unfortunately holding that one hostage)? What if I didn't want to do more soul-crushing server work (and you can bet I fucking didn't), especially since I didn't feel like I should have to since I already put massive effort into establishing my perfect setup? And even if all of that didn't apply, I still cannot recover everything completely to this day as I am missing some scripts, keys, configs, etc. that I didn't manage to backup (again, the downtime was supposed to be only about 24h, so I didn't feel like I needed it; I'm now smarter about these things obviously).
Also consider how - aside from myself - this incident had also hurt all the thousands few people who had accounts on my XMPP server. Imagine the broken friendships or marriages, lost business opportunities, relatives claimed by the reaper before they could say the final goodbye...Seriously, do these people not understand how much harm is caused by a (mildly) popular site and chat going down for weeks? People rely on them for tech support, having fun, keeping up social relations, or even mental health support and suicide prevention (this is something some have admitted to me). I can't believe this is so hard to get, therefore must admit they just don't give a shit. Malice should always be assumed if a fuckup is serious enough and there is no indication that the perpetrator cares about it (diggy's razor ^_^). Or in other words, "if it walks like evil and quacks like evil, it is fucking evil". In this case it's not even that big of a stretch, as Incognet keeps subjecting their "customers" to a portal that logs them out every few minutes (torture) and has so much contempt for them that they don't even accept E-mail communication (just so they can torture them with the portal) - which is something I've never seen in any other host. Don't you think that if they cared, they wouldn't pull such crap?
Yet, these situations keep happening because business exists to fleece you out of money, and not to "provide a service". The more corners they can cut while still not making the violation(s) so obvious and egregious that people pack it up, the better! What makes the current situation particularly bad is that modern people can be brutally tortured and yet still keep justifying their commercial oppressor (because of decades long capitalist / libertarian propaganda, according to which the business Gods graciously provide their "customers" with access to resources they "own", and thus deserve their worship); the acceptable abuse threshhold has been sent to the moon. The treatment can of course hugely differ, but still - don't expect friendly relations with a business! Don't let one pull a rug from under you - always have another host (or even two) at the ready. And pack your bags at the slightest hint of abuse; be the hero that reverses the current trend! Let's actually make the "free market" work at least somewhat close to the way its proponents say it does ("bad" business going down for the "good" to replace them).
I will lead the way. Even though it is very inconvenient to me, I don't want to deal with this trash of a host anymore, and am giving them up completely (sacrificing 200 euro in the process). I recommend you do the same, as well. After all - in a sea of VPSes guaranteeing 99+% uptime - why bother with this sham, that doesn't even bother to fulfill its primary purpose? And I was doing relatively fine on Privex (which didn't support Slackware, but was still better than this) earlier, and could have avoided this fiasco, but I guess "distrohopping syndrome" (VPS edition) took over me and I was compelled to explore. At least, now you get this exposition out of it.
And for honesty's (something totally foreign to Incognet) and thoroughness' sake, this host isn't totally worthless. After all, they still allow anonymous signup, and are pretty free speech focused. You can also upload a custom ISO that actually works. But these days, these features are also provided by many other services. And as we saw here, they are not the only ones that matter and can be invalidated by the others. Let me list them here, just in case you can't be bothered to read my entire ramblings (consider this the TL;DR version of the report):
Coming back to the vuln, a fresh Slackware 15 install on BuyVM found these (among many others [465 JPGs and 86149 PNGs]):
Still think it's a nothingburger? In my home installation of Slackware, photorec found only a little over 3k PNGs - and that's including all the random internet images, XMPP avatars, stuff I got from physical media, etc. that I've been accumulating for years. Knowing this, does it really seem so easy for 80k recoverable images to have somehow ended up in a fresh OS? Since I obviously didn't run browsers or put physical media inside that server (one without XMPP, BTW).
I actually have no idea where exactly this random stuff is coming from, but the categorical denials that it could be a vuln and insane accusations of how I supposedly uploaded them myself are quite...revealing. In the end, my advice is still to avoid uploading whatever you don't want others to see on probably any VPS until this situation is fully resolved. It is still a possibility that those files are coming from other users on the VPS, or maybe from the admin's server, or...who knows, maybe it's aliens, but I surely did not put them there myself! What I suspect (and am worried the most about) is that someone might be able to buy a VPS in the same location you are, and just lift your files, even ones deleted years ago.
UPDATE: someone who claimed to be an expert on this topic, messaged me a few weeks back and said that this can indeed happen, and has happened before even with big hosts. He gave this as a reference. So, despite widespread denials and mockery by Incognet and others, even in my MUC, it seems I might have been right after all and some hosts are still not "cleaning disks" properly. At least, it is a possible explanation, and some did not want to accept it even as that. However, it doesn't even matter at this point. Even if the data isn't actually accesible to random users of the same VPS, the vuln exploration episode revealed a lot about Incognet to me. So I consider it a success regardless.