archive snowblossom slack mining
prev 2018-06-27 next
2018-06-27 12:43:06
Shoots
this guy is crushing it, must be spending some big $
2018-06-27 12:43:10
Shoots
snow:0quas9p42rwt5vp4j92u9up3h9hjlna8fm5rah4k
2018-06-27 15:20:32
Shoots
When setting the memfield cache size is this only for the memfield? Or is this for the entire miner and all the other stuff it puts into memory?
2018-06-27 15:30:41
Rotonen
only for the memfield
2018-06-27 15:35:47
Shoots
for some reason it keeps stalling when loading into mem
2018-06-27 15:35:58
Shoots
and when I check htop I see this
2018-06-27 15:37:17
Shoots
heapdumpoutofmemory
2018-06-27 15:37:33
Shoots
even though I have xmx set to 18gb
2018-06-27 15:48:01
Rotonen
what do you have the precache set to?
2018-06-27 15:48:32
Rotonen
and, as said, get it to start with just one thread at first
2018-06-27 15:48:45
Rotonen
one problem at a time
2018-06-27 15:49:27
Rotonen
precache 1, threads 1
2018-06-27 15:49:37
Rotonen
then keep bumping the precache
2018-06-27 15:49:47
Rotonen
then start bumping the thread count
2018-06-27 16:12:49
Shoots
same problem
2018-06-27 16:12:57
Shoots
one thread one gb cache
2018-06-27 16:15:39
Tyler Boone
looks like it's taking a long time to load 1gb...
2018-06-27 16:17:04
Shoots
this is what Im using to launch ./bazel-bin/SnowBlossomMiner --jvm_flags=-Xmx185g /var/snowblossom/configs/miner-mainnet.conf
2018-06-27 16:18:12
Shoots
and I keep seeing this outofmemoryerror in my running PID's
image.png
2018-06-27 16:19:22
Tyler Boone
I don't understand that last statement
2018-06-27 16:20:13
Shoots
its going now with 1 thread and 1gb loaded into mem, you were right just took a long time
2018-06-27 16:23:03
Shoots
it would be nice if it wasnt trying to load work from the pool while its still loading the memcache
2018-06-27 16:23:11
Tyler Boone
I've noticed on my machine once it gets to about 47gb loaded it hangs for a good minute or two. I haven't looked into it because it doesn't bother me, but I suppose it's some heavy lifting inside the garbage collector
2018-06-27 16:23:48
Tyler Boone
meh... all the threads are blocked waiting, so I don't see a problem
2018-06-27 16:25:27
Shoots
I just mean so you dont see it writing out that its stalled
2018-06-27 16:25:51
Tyler Boone
oh yeah
2018-06-27 16:27:19
Tyler Boone
I actually did want to load it before starting all that nonsense, but because of the design of the FieldScan to support swapping to a higher snowfield without restarting the miner it would have required a lot more piping and I didn't want to fuddle up the code too bad
2018-06-27 16:27:58
Shoots
also the solo miner seems to load faster than the pool miner
2018-06-27 16:29:19
Tyler Boone
!
2018-06-27 16:29:25
Tyler Boone
how many times did you repro that?
2018-06-27 16:29:55
Shoots
well Im just noticing by the approx ten times Ive tried
2018-06-27 16:30:10
Tyler Boone
no clue.
2018-06-27 16:30:28
Shoots
the solo miner will get the first 6gb in before the first stall message, the pool miner takes ten minutes before it gets even a few gb loaded
2018-06-27 16:30:49
Shoots
hard to explain
2018-06-27 16:36:19
Rotonen
i’m also +1 on not disk mining before memfield or precache is done - the disk mining saturates io and there is a net loss
2018-06-27 16:37:37
Tyler Boone
there shouldn't be any disk mining happening before precache
2018-06-27 16:37:43
Shoots
13 min with solo miner now and only 5gb loaded
2018-06-27 16:37:52
Tyler Boone
wtf?
2018-06-27 16:38:03
Shoots
pool miner I mean
2018-06-27 16:38:04
Tyler Boone
51gb loads for me in like 3-4 minutes
2018-06-27 16:38:13
Shoots
its as if the work from the pool is interrupting it
2018-06-27 16:39:18
Tyler Boone
all the threads should be blocked on that line (except the one loading the precache)
2018-06-27 16:39:51
Tyler Boone
you can do [kill -3 <pid>] to force java to print out the stacktraces
2018-06-27 16:40:19
Tyler Boone
(or use jstack, whichever you prefer)
2018-06-27 16:40:51
Rotonen
@Tyler Boone it’s been definitively mining for me before memfield is complete
2018-06-27 16:41:17
Tyler Boone
@Rotonen, send logs
2018-06-27 16:41:36
Rotonen
or does it start mining the early loaded memfield?
2018-06-27 16:41:45
Shoots
when using memfield (non hybrid) it mines for me before done loading too
2018-06-27 16:41:49
Tyler Boone
no, it waits for the entire precache
2018-06-27 16:42:06
Fireduck
rtfc
2018-06-27 16:42:09
Shoots
when hybrid mining it doesnt have a hashrate until the cache is done loading
2018-06-27 16:42:37
Rotonen
@Tyler Boone will do, not near a privileged terminal within the next 6h or so, though
2018-06-27 16:42:46
Shoots
but its taking a god awful long time to load the cache into mem
2018-06-27 16:43:56
Fireduck
@Shoots sure it is the same snowpath between solo and pool?
2018-06-27 16:44:16
Tyler Boone
maybe the drive is sleepy
2018-06-27 16:45:02
Rotonen
sustaining io on cloud offerings usually gets one throttled
2018-06-27 16:45:03
Shoots
yup
2018-06-27 16:45:22
Shoots
using same config just commenting out the node_host and pool_host when I switch
2018-06-27 16:48:16
Shoots
over 25 min now INFO: pre-caching snowfield: loaded 7 gb of 102 (6%)
2018-06-27 16:48:51
Shoots
normal memfield mining it takes about 10 minutes to load 185gb and use all of my memory
2018-06-27 16:53:21
Shoots
ok there we go, I was using PoolMiner that was built with bazel build :all, if I build and use PoolMiner_deploy.jar it loads faster
2018-06-27 16:53:40
Fireduck
strange
2018-06-27 16:53:51
Shoots
that was with a fresh git pull
2018-06-27 16:54:59
Rotonen
i also have much better performance with the deployment builds
2018-06-27 16:55:15
Rotonen
that’s not expected?
2018-06-27 16:55:55
Fireduck
not at all
2018-06-27 16:56:09
Fireduck
should be no difference other than maybe a few hundred ms at startup
2018-06-27 16:56:24
Fireduck
unless someone fucked up the code in master
2018-06-27 16:56:33
Rotonen
implicit jvm flags via bazel? it’s a weird target class
2018-06-27 16:56:47
Fireduck
@Shoots if you want to pull down the 1.1.2 release tag and build that, I'd appreciate that
2018-06-27 16:56:51
Fireduck
it could be a performance regression
2018-06-27 16:57:29
Fireduck
@Rotonen good point. in that case, I'd expect better via bazel-bin/PoolMiner
2018-06-27 16:57:33
Rotonen
he was also noting earlier he had the best performance with 1.0.7
2018-06-27 16:58:29
Fireduck
yeah, might be a regression
2018-06-27 16:58:44
Fireduck
that entire thing needs to be restructured
2018-06-27 16:58:53
Fireduck
less emphasis on smoothly handling field changes
2018-06-27 16:58:56
Rotonen
i’ll do a benchmark sweep across tags later
2018-06-27 16:59:12
Fireduck
more emphasis on allowing field peices to have various optimization/buffer/behavior on or off
2018-06-27 16:59:19
Fireduck
@Rotonen I love you
2018-06-27 16:59:26
Rotonen
i’d just drop the whole field change stuff
2018-06-27 16:59:43
Shoots
am I not pulling 1.1.2 when I do git pull?
2018-06-27 16:59:45
Rotonen
the battery of hits and misses confuses people
2018-06-27 16:59:49
Fireduck
or at least kick it out to a base layer and restart everything
2018-06-27 18:36:45
Shoots
1:20 min only 43gb loaded into memory
2018-06-27 18:37:18
Shoots
it started faster, but as soon as it started printing out that it was stalled it slowed right down
2018-06-27 18:44:46
Shoots
Just downloaded the 1.12 release and its loading fine, so it must be something to do with the version I was building
2018-06-27 19:16:32
Shoots
ok so now I just want to test to see if this version will go over my max amount of memory which is 185gb, but Im getting out of memory error when it hits 137gb.
2018-06-27 19:16:33
Shoots
java -Xmx185g -jar bazel-bin/PoolMiner_deploy.jar /var/snowblossom/configs/miner-mainnet.conf
2018-06-27 19:34:07
Rotonen
define ’out of memory error’
2018-06-27 19:34:09
Shoots
yeah super weird
2018-06-27 19:34:10
Shoots
Jun 27, 2018 7:33:01 PM snowblossom.miner.PoolMiner$MinerThread run WARNING: Error: java.lang.OutOfMemoryError: Java heap space
2018-06-27 19:34:33
Shoots
only 137/185gb used, and Im using xmx flag to set the java limit higher
2018-06-27 19:34:46
Rotonen
try with a matching -Xms
2018-06-27 19:35:10
Rotonen
do also note all java objects have overhead
2018-06-27 19:35:20
Shoots
yes
2018-06-27 19:35:29
Shoots
but look at the htop screenshot
2018-06-27 19:35:34
Shoots
far from using the 185gb
2018-06-27 19:35:56
Shoots
never had this problem with other version, only since I built :all with 1.1.2 source
2018-06-27 19:36:02
Clueless
who's the crazy 4hk mining guy?
2018-06-27 19:36:37
Rotonen
@Shoots i do not fully trust accounting from htop, use top ot atop
2018-06-27 19:37:07
Shoots
htop has been accurate for me until now, Ill check top
2018-06-27 19:37:47
Rotonen
i’ve had various silly issues with htop over the past 15 years
2018-06-27 20:00:10
Shoots
well it eventually ran out of mem as well, not sure why 1.1.2 gives me so much lower hr than the older 1.0 version I was using before
2018-06-27 20:00:18
Shoots
3.4 > 2.5mh
2018-06-27 20:01:00
Fireduck
probably terrible code by drunks
2018-06-27 20:01:25
Shoots
lol
2018-06-27 20:02:02
Shoots
I have both the old version and the new version built on this same vm, I can run both and I get 3.4 on one and 2.5 on the other
2018-06-27 20:02:26
Shoots
I even built 1.1.2 from source cause someone mentioned doing that would help with threading
2018-06-27 20:02:28
Fireduck
and this is memory mining?
2018-06-27 20:02:32
Shoots
yes
2018-06-27 20:02:48
Fireduck
would you mine filing an issue on github?
2018-06-27 20:02:48
Shoots
Im going to try some other versions and test
2018-06-27 20:03:27
Fireduck
sounds like something got broken when the pre-cache code went in, maybe
2018-06-27 20:09:46
Shoots
Im trying 1.0.7 the first version with pre cache
2018-06-27 20:28:31
Shoots
yup 1.0.7 is slower too, 48c vm, gets 3.4mh on 1.0.6 and only 2.5mh on 1.0.7+
2018-06-27 20:29:07
Shoots
thats using the poolminer
2018-06-27 20:29:15
Fireduck
sure
2018-06-27 23:08:35
stoner19
@Shoots whats your hardware to get 3.4M/s?
2018-06-27 23:08:45
stoner19
compute instance?
2018-06-27 23:08:49
Shoots
48c vm