Interesting irony trying to push faster the C++ pipeline - more to speed than appears. Spawned a $6 droplet (what DigitalOcean calls Premium AMD with NVMe) to test the C++ pipeline arm. Some numbers/thoughts, with pinch of salt given this is nowhere close to scientific an assay.
- First observation - process was 5x slower than the regular $5 droplet.
- Since the idea was to spawn a new droplet with an NVMe to see if a better write speed would help , checked hdparm. hdparm shows disk is not any better than my local SSD, weird as it might be. [1]
- Bumped the box up to a 2 vCPU, 4GB state but weirder enough, numbers didn't drop one bit.
- Au contraire, the 1GB droplet runs at a steady sate of 85% CPU utilisation. This process itself, peaks at about 65-70%. Memory is an interesting metric since the data sets don't have a fixed size. It would be interesting to see how the memory is being stacked up. [2]
One experiment done, now two more to take a look at, to learn and improve things if at all it were possible. Thus far, an interesting run.
[1] hdparm numbers - The WDS240G2G0A SATA SSD is about as fast as the droplets' NVMe. Should follow this up with storage/CPU optimized droplets.
Local SSD
/dev/sda:
Timing cached reads: 10350 MB in 1.99 seconds = 5191.89 MB/sec
Timing buffered disk reads: 1036 MB in 3.00 seconds = 345.21 MB/sec```AMD NVMe
/dev/vda:
Timing cached reads: 13564 MB in 2.00 seconds = 6790.73 MB/sec
HDIODRIVECMD(identify) failed: Inappropriate ioctl for device
Timing buffered disk reads: 3934 MB in 3.00 seconds = 1311.27 MB/sec```Regular Intel+SSD
Timing cached reads: 5208 MB in 2.00 seconds = 2605.95 MB/sec
HDIODRIVECMD(identify) failed: Inappropriate ioctl for device
Timing buffered disk reads: 610 MB in 3.00 seconds = 203.19 MB/sec
[2] I'm not very sure if the upfront allocation of a 4096 struct array does as much. That would also be a place to restructure the concern of this arm of the pipeline.