CounterParty server database is behind backend

Hello,

my CounterParty server stops working and all RPC calls returning error -3200: “Counterparty database is behind backend.”. The bitcoind log shows it’s syncing without isses with the Bitcoin network.

2016-03-02 01:27:14 UpdateTip: new best=000000000000000004400bf10cd48e145fb704e49849ea98f20f37788db640a6  height=400755  log2_work=84.217476  tx=113723778  date=2016-03-02 01:26:05 progress=0.999999  cache=0.5MiB(0tx)

Is it normal for CounterParty server to get stuck for over 20 mins?

2016-03-02-T01:55:42+0100 [INFO] Block: 400749 (7.93s, hashes: L:1da5c / TX:eea41 / M:32875)
2016-03-02-T01:59:09+0100 [INFO] Mempool cache initialized: 207.15s for 20,000 transactions
2016-03-02-T01:59:09+0100 [INFO] Ready for queries.

Now stuck for 30 mins

EDIT: After restarting the server it catched up with the blocks but now it is stuck again (after mempool cache is initialized)… Last messages:

2016-03-02-T02:53:22+0100 [INFO] Block: 400758 (1.74s, hashes: L:31a83 / TX:f4129 / M:fc7c9)
2016-03-02-T02:53:24+0100 [INFO] Block: 400759 (1.98s, hashes: L:d00fa / TX:b5917 / M:939ec)
2016-03-02-T02:53:32+0100 [INFO] Block: 400760 (7.85s, hashes: L:70232 / TX:c554a / M:7dc24)
2016-03-02-T02:55:56+0100 [INFO] Mempool cache initialized: 144.55s for 20,000 transactions
2016-03-02-T02:55:57+0100 [INFO] Ready for queries.

And next block already received by bitcoind but not being processed by CounterParty server…

016-03-02 01:53:54 UpdateTip: new best=0000000000000000017536aa425e312542a9d8e0b9724deda342f08aa1a7e7b1  height=400761  log2_work=84.217746  tx=113731736  date=2016-03-02 01:52:14 progress=0.999999  cache=56.9MiB(7381tx)

EDIT2: Apparently, it parsed the following block much later…

2016-03-02-T03:08:57+0100 [INFO] Block: 400761 (25.49s, hashes: L:2bd54 / TX:bfce3 / M:9d804)

See a graph of recent number transactions (eg here: I sent BTC/XCP/TOKEN a while ago, but the transaction hasn’t gone through - why?), that’s why.

In recent months this happens frequently because there are so many transactions. It’s recommended to put Blockchain indexes ( if your filesystem allows you to do that; if not then the entire blockchain) and Counterparty DB on SSD). That makes these problems less frequent.

A bit more context, as I’ve been doing some performance characterization and optimization in spare time…

Notes (these numbers correspond to markings on the screenshot):

  1. When a new block is created, indexing kicks in. That’s a write-heavy activity. You can see at the same time there’s no significant network traffic - it’s all internal stuff.
  2. In order to create indexes, Bitcoin Core must read its data, so there’s a lot of reading going on too.
  3. You can’t expect more than 150-200 IO (combined reads + writes) from a HDD so when you get over 150, you can see timeouts. As you can see here, it goes to 250 sometimes. But the entire time in large rectangles was spent in Counterparty Server RPC timeouts. (Edit: 1 row = 1 second; so Counterparty Server didn’t quit .)

Note that Counterparty Server runs on a different system, which makes performance troubleshooting easier. Its performance in this case isn’t an issue (although it too would benefit from SSD) so I’ll just show another screenshot from Bitcoin Server when times are good (i.e. no timeouts on RPC client):

  1. Plenty of reads, because Counterparty Server is catching up after timeouts
    Plenty of writes, as well, but only 15-20 per second, vs. 20-40 before (the second screenshot was taken 5 mins after the first).
  2. You can see bursts of 2 MB/s going out to the network. PktIn is regular Bitcoin network and whatnot incoming traffic, not different from the first screenshot .
    These outgoing batches of data is what Bitcoin Core sends to RPC client (Counterparty Server). The size of these can vary and at least partially depends on rpc-batch-size = in Counterparty’s server.conf. In this particular case I have rpc-batch-size = 500 which seems a bit too much for this particular setup (i.e. slow HDD on the system running Bitcoin Core). The default is 20 and it seems 200 was causing fewer timeouts during busy times on Bitcoin Core. When I used rpc-batch-size = 100 then batches of network sends were smaller (as one might expect). And in bitcoin.conf you can also rpcthreads and rpctimeout (the first one is pointless to increase if your server can’t take it; the second one maybe can allow the server to tolerate longer timeouts).

I also looked into the internals of Bitcoin Core and Counterparty Server performance in terms of I/O and while I haven’t processed most of data I collected, I seems to me that LevelDB (txindex, specifically) is what causes a lot of IO on Bitcoin Core, and haven’t found a way to make it run faster. Apart from the obvious, of course, which is to increase dbcache to a higher value if you have more RAM.

Realistically, though, you need SSD for Bitcoin Core and if you have more than 4 GB of RAM then you can put Counterparty Server on the same system. Below that and/or without SSD, it’s tough.

  1. Bitcoin Core and Counterparty Server on the same system (screenshot below) can increase bandwidth consumption
  2. More importantly, though, one doesn’t have to wait long enough to see (Reads + Writes) hit 200 or even more (over 300, in the screenshot, and I waited just 2-3 minutes). At this point without SSD you’d likely get I/O timeouts in Counterparty and your app. Maybe not immediately as this was just a spiike, but pretty soon.

With SSD you can go to few thousand IOPS and not only avoid timeouts, but also recover from them (if they happen) without Counterparty Server timing out 12 times and then quitting (stopping).
When the blockchain gets 2-3 new blocks within 15 mins, or a 2-3 block reorganization (similar effect, essentially) a lot of churning has to be done, and a single HDD can’t handle that in a timely manner and still respond to RPC requests by Counterparty Server.

##Mempool Cache

You can see you loaded 20,000 transactions from the mempool cache, that’s a lot. In July 2015 (https://en.bitcoin.it/wiki/July_2015_flood_attack) the attack loaded 80K.

At time of attack you can’t do much even with faster h/w. But at least have this in bitcoin.conf:

minrelaytxfee = 0.00005
limitfreerelay = 0

In Bitcoin Core 0.12 (you can get a release candidate with addrindex from BTCDrak’s repo now - I’ve been running it for a few days, it seems good enough for test/dev) this will be handled better, and these settings won’t be necessary - see https://gist.github.com/laanwj/efe29c7661ce9b6620a7.

Thank you very much for your detailed answer.

Today I’m not experiencing this error anymore, so probably yesterday I had other processes in background stealing CPU/IO resources or the Bitcoin spam attack has slowed down now.
I have just added rpc-batch-size = 150 on my server.conf. I already had configured Bitcoin core with:

minrelaytxfee = 0.00005
limitfreerelay = 0

Anyway, looks like I underestimated the resources (I/O and CPU) needed to run several crypto wallets on the same server…

You’re welcome.

Today the TX situation is even worse - a new record it seems, over 257,000 transactions: https://blockchain.info/charts/n-transactions

If this is a test server where you can experiment, check out 0.12, it seems more responsive to me. You can install the both (i.e. get the binaries and put them in a different path) and run either 0.11.2 or 0.12rc* on the same blockchain (not at the same time, of course).