archive snowblossom slack mining
prev 2018-06-25 next
2018-06-25 01:52:01
Kayla
alrighty, no clue if that's slow, average or wut but ```[2018-06-25 01:50:39] INFO snowblossom.miner.PoolMiner printStats 1-min: 5.760K/s, 5-min: 5.758K/s, hour: 693.236/s [2018-06-25 01:50:39] INFO snowblossom.miner.PoolMiner printStats Shares: 1 (rejected 0) (blocks 0) [2018-06-25 01:50:54] INFO snowblossom.miner.PoolMiner printStats 15 Second mining rate: 5752.999/sec - at this rate 12.151 minutes per share (diff 22.000) [2018-06-25 01:50:54] INFO snowblossom.miner.PoolMiner printStats 1-min: 5.743K/s, 5-min: 5.762K/s, hour: 717.213/s [2018-06-25 01:50:54] INFO snowblossom.miner.PoolMiner printStats Shares: 1 (rejected 0) (blocks 0)``` took me like 3 days to finally have it running (with the torrenting thingy, raid 0 setup and all xD)
2018-06-25 02:34:14
stoner19
no blocks all day at snowday today. ouch.
2018-06-25 02:34:45
Fireduck
blocks are no longer cool
2018-06-25 02:36:36
Kayla
is that due to the difficulty staying the same while nethash dropped by half (with the moving from field 6 to 7) ? _idk how it works just yet, am still noob_ :stuck_out_tongue:
2018-06-25 02:40:14
stoner19
blocks are are and always will be cool
2018-06-25 02:40:23
stoner19
its probably because of miners jumping around from pool to pool.
2018-06-25 04:09:49
Clueless
Yeah, I noticed
2018-06-25 04:10:00
Clueless
I get like 10MH/s on snowday, then they leave for some reason
2018-06-25 04:18:31
Fireduck
Ok, at this point I'd really like people to vote yes on prop 2.
2018-06-25 04:18:39
Fireduck
It gives school teachers a raise or something.
2018-06-25 04:19:27
Ya_Mon
I would agree to give teacher a raise if they get rid of that "untouchable after 10 years, bitches" rule
2018-06-25 04:20:32
Ya_Mon
better yet, fuck public schools, either private or homeschooling.
2018-06-25 04:31:14
Protovist
I'm not so sure about prop 2... it essentially removes the hashrate requirement to advance snowfields.
2018-06-25 04:32:38
Clueless
@Protovist ? I don't think so.
2018-06-25 04:33:40
Protovist
Once someone is creating blocks with a higher snowfield, the weighting means that only blocks from that field will be added to the main chain.
2018-06-25 04:34:28
Fireduck
Weight only comes from required field
2018-06-25 04:34:37
Fireduck
Not whatever someone mines with
2018-06-25 04:40:25
Fireduck
Someone can mine with field 11 and it makes no difference. Only the current active field matters for weight. Which continues to only be triggered by difficulty
2018-06-25 04:40:54
Fireduck
The work sum weighting had no effect on difficulty
2018-06-25 04:40:58
complexring
can difficulty drop?
2018-06-25 04:41:05
Fireduck
Yes
2018-06-25 04:41:15
Fireduck
But activated field can not
2018-06-25 04:41:22
complexring
awww
2018-06-25 04:41:25
complexring
ok
2018-06-25 04:41:27
complexring
got it
2018-06-25 04:42:30
Clueless
snowday needs more power!
2018-06-25 04:42:38
Clueless
I'm givin it all she's got captain!
2018-06-25 04:56:44
Fireduck
@Protovist I appreciate the scrutiny, but I think in this case there is nothing worry about. The fields activate with difficulty which adjusts based on hash rate. That all stays the same. The only difference is that once the chain activates a higher field, the chain with the higher field has an advantage in work sum weighting, which hopefully counter acts the hash rate drop when going to a higher field.
2018-06-25 04:57:15
Protovist
That makes sense. As long as it's the activated field, my concern is void.
2018-06-25 04:57:58
Fireduck
right. The weighting is based on activated field, not which field was used for a particular block.
2018-06-25 15:08:32
Rotonen
@Shoots did you try the amazon skylake xeons? at least the ones on digital ocean were rather performant, but i suspect there is less resource contention on DO than on AWS
2018-06-25 15:13:24
Shoots
I have, but they only have 192gb of mem
2018-06-25 15:13:31
Shoots
and for some reason my miner uses 210gb
2018-06-25 15:13:51
Shoots
unless I run the latest version of the miner, then it maxes out the mem usage at 145gb, but the hr is much lower
2018-06-25 15:14:11
Shoots
and on top of that there's no available spots anymore
2018-06-25 15:37:43
stoner19
who is running Vauxhall? Would be nice to see miner list, pool hash, net hash, and time since last block on the website
2018-06-25 15:38:37
kaka
agree
2018-06-25 15:45:36
Rotonen
@Shoots yeah, i suppose @Fireduck will eventually fix the miner to not require obscene amounts of JRE tweaking
2018-06-25 15:46:59
Shoots
yeah tweaking the java parameters is beyond me
2018-06-25 18:48:45
stoner19
2% fee on protopool? Surprised there are that many miners for it
2018-06-25 19:06:19
Rotonen
block density matters more early on
2018-06-25 19:53:54
fydel
INFO: Send new work to 13 workers. Keeping 10, Dropping 3
2018-06-25 19:54:07
fydel
what does the "Dropping" mean?
2018-06-25 19:54:54
fydel
i did have have it yesterday. but now i have 1-5 dropping in nearly every message.
2018-06-25 19:55:22
fydel
network issue?
2018-06-25 19:56:02
Rotonen
look at the code, probably something like ’did not hear back from it in a while’
2018-06-25 19:58:48
fydel
drop_count++; //logger.info("Error in send work: " + t);
2018-06-25 20:00:15
fydel
rejected shares?
2018-06-25 20:03:00
fydel
INFO: Work block load error: java.lang.RuntimeException: Unable to select a field of at least 7
2018-06-25 20:03:12
fydel
ah. damn. i think my snowfield is broken.
2018-06-25 20:16:54
Rotonen
curious, so definitely not just an old field?
2018-06-25 20:23:04
fydel
no. after closer inspection it was a wrong sym link.
2018-06-25 20:39:49
Shoots
if I create a swap file in ubuntu will the miner use that without enabling hybrid mining?
2018-06-25 20:39:51
Fireduck
drop_count about clients that are removed due to the links already being broken
2018-06-25 20:40:00
Fireduck
noted by trying to send them work resulting in errors
2018-06-25 20:40:10
Fireduck
so it means they have disconnected and are gone
2018-06-25 20:40:22
Fireduck
but grpc doesn't really tell us that until we try to send something
2018-06-25 21:07:11
Rotonen
@Shoots see vm.swappiness
2018-06-25 21:35:32
Shoots
swap file seems to lock up the vm
2018-06-25 21:55:22
Shoots
im trying hybrid mining now
2018-06-25 21:55:50
Shoots
I really want to get this m5d instance running, it has the highest hash per $, but has a tad too little of memory
2018-06-25 21:56:04
Fireduck
how much does it have?
2018-06-25 21:56:09
Shoots
192
2018-06-25 21:56:18
Shoots
my miner seem to use about 210gb
2018-06-25 21:56:22
Fireduck
strange
2018-06-25 21:56:50
Shoots
when I installed java 10 and the newest miner last night it was using only 145gb
2018-06-25 21:56:59
Shoots
but something wasnt right cause the hr was way too low
2018-06-25 21:57:58
Shoots
now Ive compiled from source with openjdk and trying hybrid mining with 180gb set as my cache size
2018-06-25 22:01:35
Rotonen
@Shoots if you're like 10G short, try allocating 50G to zram
2018-06-25 22:04:10
Rotonen
@Shoots debian 9, default-jre-8-headless, and i'm getting by 135G ram use for memfield (though bumped -Xmx to 200G as why not, rather have it fire the GC less often)
2018-06-25 22:04:57
Rotonen
@Shoots in a nutshell zram is a way of trading spare cpu cycles for 'more ram' - even with ram mining snowblossom is not actually cpu limited
2018-06-25 22:09:38
Shoots
I wonder if the number of cores or threads impacts the ram usage?
2018-06-25 22:11:25
Rotonen
it does
2018-06-25 22:12:06
Rotonen
try to find the minimal thread count where the next one over does not bring you any meaningful extra
2018-06-25 22:12:44
Shoots
Oh ok
2018-06-25 22:12:54
Shoots
Its a 48c VM I have it set to 96
2018-06-25 22:12:57
Rotonen
start with 1, check where the ram use is after 1min and 5min are ~equal, double, repeat until gain is ~naught
2018-06-25 22:13:22
Rotonen
those are vcores so 48 or 24 more plausible sane thread counts
2018-06-25 22:13:28
Shoots
Getting 0h with hybrid miner
2018-06-25 22:13:44
Rotonen
you have 48 potentially-shared-with-other-guests hyperthreads from the hypervisor
2018-06-25 22:13:57
Shoots
Yeah
2018-06-25 22:14:23
Rotonen
but try memfield and 12, 24, 48
2018-06-25 22:14:33
Rotonen
start with 12, to see if you get it going at all
2018-06-25 22:15:06
Rotonen
usually in high performance computing contexts `cloud vcore count / 4` ~ `real cpu count equivalent`
2018-06-25 22:16:12
Shoots
why am I not hashing I wonder
2018-06-25 22:16:16
Rotonen
but borderline impossible to guess how the memory channels are provisioned for amazon guests, if they do certain things not optimally your snowblossom performance can actually be up to dumb luck in how it just happened to map when provisioned (or even based on hypervisor load)
2018-06-25 22:16:24
Rotonen
well that's just a miner bug, i'd say
2018-06-25 22:16:36
Shoots
yeah nothing to worry about lol
2018-06-25 22:16:52
Shoots
it seemed to cap out at 138gb or ram usage
2018-06-25 22:16:54
Shoots
but 0h
2018-06-25 22:17:16
Rotonen
with what exact config?
2018-06-25 22:17:37
Shoots
# network # (snowblososm/mainnet, teapot/testnet, spoon/regtest) network=snowblossom #node_host= pool_host=http://snow.protopool.io #node_port=23380 #pool_host=http://pool.snowblossom.cluelessperson.com # the location of "snow" fields for mining snow_path=snow/mainnet # automatically generate mining snow files. # CAUTION! INTENSIVE! You may wish to torrent instead. # torrents: https://snowblossom.org/snowfields/index.html #auto_snow=true # pick an address (at random for now)from this wallet to mine to #mine_to_wallet=wallets/mainnet # or mine to address mine_to_address=snow: # number of cpu threads to commit to PoW threads=96 memfield=true memfield_precache_gb=180
2018-06-25 22:17:56
Rotonen
i'm running fine with ``` memfield=true mine_to_address=<redacted> network=snowblossom pool_host=http://snowplough.kekku.li snow_path=snow/mainnet threads=24 ```
2018-06-25 22:18:02
Rotonen
~135Gb ram use
2018-06-25 22:18:11
Rotonen
6 core E5
2018-06-25 22:18:17
Shoots
whats your hr with 24 threads?
2018-06-25 22:18:30
Rotonen
850k, which is what i'm expecting as well
2018-06-25 22:18:48
Shoots
thats low compared to what I get though, with 24 threads I get about 1.2mh
2018-06-25 22:18:58
Shoots
24cores I should say
2018-06-25 22:19:01
Rotonen
i had a script iterate from 1 to 128 threads and 24 was the sweet spot
2018-06-25 22:19:28
Rotonen
``` $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Stepping: 2 CPU MHz: 3599.975 CPU max MHz: 3800,0000 CPU min MHz: 1200,0000 BogoMIPS: 6983.10 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 15360K NUMA node0 CPU(s): 0-11 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts ``` older hardware, in line with what i expect
2018-06-25 22:20:15
Shoots
oh its 12 cores, thats pretty good actually for 12 cores
2018-06-25 22:20:22
Shoots
thats your own hardware?
2018-06-25 22:20:22
Rotonen
6 cores
2018-06-25 22:20:33
Shoots
well thats really good for 6 cores damn
2018-06-25 22:20:52
Rotonen
it's not the cores, it's not the memory channels, it's the ratio of cores to memory channels
2018-06-25 22:20:57
Rotonen
i have 6 cores over 4 memory channels
2018-06-25 22:21:23
Rotonen
then the generational gains are like 5% per generation for the processor cores and about 3% for memory
2018-06-25 22:21:40
Rotonen
so i could get 1M on a modern xeon and something like 2,5M on an AMD EPYC
2018-06-25 22:22:00
Rotonen
potentially 1,6M on the new i9-esque 6 channel stuff
2018-06-25 22:22:24
Rotonen
so if you find an old intel E7 for cheap, give that a spin, that should always land you north of 1M
2018-06-25 22:23:17
Rotonen
but those processors usually cost tens of thousands when new, so they're probably going to be used to the bitter end by whomever needs to buy them (and they're actually usually bought more for the reliability than the throughput)
2018-06-25 22:24:03
Rotonen
https://ark.intel.com/products/82765/Intel-Xeon-Processor-E5-1650-v3-15M-Cache-3_50-GHz Intel® Xeon® Processor E5-1650 v3 (15M Cache, 3.50 GHz) quick reference guide including specifications, features, pricing, compatibility, design documentation, ordering codes, spec codes and more.
2018-06-25 22:24:19
Rotonen
and this processor is from 2014
2018-06-25 22:24:36
Rotonen
old hardware is rather viable
2018-06-25 22:26:40
Rotonen
@Shoots but i'm curious as to what you get with 12 threads and memfield
2018-06-25 22:27:00
Rotonen
@Shoots that's what i'd start with given the vcores / 4 shorthand rule
2018-06-25 22:28:53
Shoots
what do you mean by shorthand rule?
2018-06-25 22:29:00
Shoots
just by using 12 threads?
2018-06-25 22:29:28
Rotonen
no, if you use any cloud services for anything performance critical and you compare to real hardware, dividing the cloud 'core count' by four makes it more apples to apples in what one can expect
2018-06-25 22:30:04
Shoots
well Im getting 3.4mh with 96 threads right before it runs out of mem
2018-06-25 22:30:47
Rotonen
that means the backing system is a multi socket system and you're getting really lucky in how the memory of your VM is spread over 12 channels (or 24 channels if it is a quad socket system)
2018-06-25 22:31:13
Shoots
its probably cause its latest gen
2018-06-25 22:31:33
Rotonen
probably dual socket and mostly memory bandwidth idle by your neighbours, if i estimate about 300kH/s per memory channel, that's 3,6M
2018-06-25 22:32:05
Rotonen
usually the truth about hashes per memory channel maximums is somewhere between 200kH/s and 400kH/s
2018-06-25 22:32:29
Rotonen
in my 'wizard' page i'm lowballing that to 200, but i really need to format that whole thing better
2018-06-25 22:32:37
Shoots
I wonder if it will use more memory cause Im running in tmux sesssion
2018-06-25 22:32:42
Rotonen
no
2018-06-25 22:33:45
Shoots
ok cause I was using damn near 130gb with field 6
2018-06-25 22:33:54
Shoots
trying 12 threads now
2018-06-25 22:34:11
Clueless
@Shoots I could write a script that experiments with different values and charts out the most efficient settings
2018-06-25 22:34:27
Rotonen
@Clueless rather make the miner autotune the thread count at runtime
2018-06-25 22:34:59
Shoots
it would be nice if it was a separate utility or if it asked you if you wanted to run it the first time
2018-06-25 22:35:04
Rotonen
i'll actually take a stab at that next week most likely, i've not touched java since 1.4 EE was the hot new thing just out
2018-06-25 22:35:23
Rotonen
@Shoots i'd rather it always warms up the system carefully and speeds up slow
2018-06-25 22:35:42
Rotonen
@Shoots not starting from 1 thread, but from a sensible ballpark, like the system thread count
2018-06-25 22:35:54
Rotonen
or quarter that and then doubling until matching and then trying +50% or somesuch
2018-06-25 22:36:04
Rotonen
if i actually bother, can figure it out as i go
2018-06-25 22:37:24
Rotonen
the most naive approach is of course to just let it bump the thead count up one at a time, but that'll take a very long time to get hot on large systems
2018-06-25 22:37:31
Rotonen
like 3 hours or so
2018-06-25 22:38:05
Rotonen
miners don't even seem to have patience for the torrenting of a snow field, memfield loading or getting the first stable 5min rate :slightly_smiling_face:
2018-06-25 22:38:44
Shoots
it would need to use a lower field to do it faster
2018-06-25 22:39:36
Rotonen
i dream of the fields being split into 1G files which can be shoved into ram on demand until hitting the heap limit (and a script which sets the heap limit upon launch to be a bit shy of the available ram)
2018-06-25 22:39:50
Shoots
is there anyway I can see whats in ram and see whats eating up the extra 90gb overtop of the field?
2018-06-25 22:40:08
Rotonen
see ptrace, strace
2018-06-25 22:41:53
Shoots
strace -p 2031 strace: Process 2031 attached futex(0x7f40126309d0, FUTEX_WAIT, 2033, NULL
2018-06-25 22:43:22
Rotonen
seems there is a java specific one: https://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html that should feed into any vm visualizers
2018-06-25 22:48:26
Shoots
dumping now
2018-06-25 22:48:31
Shoots
from a different vm
2018-06-25 22:48:37
Shoots
12 threads already up to 150gb
2018-06-25 22:50:37
Rotonen
``` PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7380 root 35 15 0,206t 0,136t 17820 S 1180 55,5 27:04.99 java -XX:+UseParallelOldGC -Xmx200G -jar PoolMiner_deploy.jar miner.conf ```
2018-06-25 22:51:11
Rotonen
what number do you track? RES is the one which matters
2018-06-25 22:51:40
Shoots
im going to try your command
2018-06-25 22:52:04
Rotonen
the parallel old gc, when used alone, decreases performance
2018-06-25 22:52:10
Rotonen
that's just the random test run i'm on currently
2018-06-25 22:52:16
Rotonen
but skip that one
2018-06-25 22:52:30
Rotonen
(or maybe, since you have a different java, try it? dunno at this point)
2018-06-25 22:52:42
Shoots
Im running openjkd8
2018-06-25 22:52:50
Shoots
cause thats what I needed to build from source
2018-06-25 22:53:02
Rotonen
i build on a mac
2018-06-25 22:54:21
Rotonen
but on debian 9, the package `default-jre-headless` works the best for me
2018-06-25 22:54:36
Rotonen
and the best i get with that so far is `nice -n 15 java -Xmx200G -jar PoolMiner_deploy.jar miner.conf`
2018-06-25 22:55:03
Rotonen
then oracle linux with oracle java can get me more, but requires tens of parametres for the jre for ultimately very little gain
2018-06-25 22:56:20
Shoots
theres gotta be something I can do to reduce this memory issue
2018-06-25 22:56:21
Rotonen
could also be nicing the miner perversely enough gives one slight extra oompf as it gets less in the way of system things
2018-06-25 22:56:33
Shoots
any chance you could compress your dir for me?
2018-06-25 22:56:37
Shoots
and share
2018-06-25 22:57:15
Rotonen
now, this is getting into dangerous country as i could just give you any malware
2018-06-25 22:57:23
Shoots
I trust you
2018-06-25 22:58:06
Rotonen
i'm half sure it's something to do with the fact we use a different jre, though
2018-06-25 22:58:17
Rotonen
bazel should output the same `_deploy.jar`
2018-06-25 22:58:40
Shoots
I tried building with java 10 and got an error
2018-06-25 22:58:56
Shoots
uninstalled java 10 and installed openjkd and it worked fine
2018-06-25 22:59:21
Shoots
also weird that when hybrid mining I get 0h/s
2018-06-25 22:59:31
Shoots
Does it show 0 until you are fully done loading?
2018-06-25 22:59:36
Rotonen
no, it does not
2018-06-25 23:00:41
Rotonen
hmm, or at least not when memfielding, don't recall about hybrid, can try
2018-06-25 23:00:44
Shoots
mem dump to a text file used up my disk space and ran out :S
2018-06-25 23:01:05
Rotonen
well, it is an actual mem dump, that's the full memory content
2018-06-25 23:01:23
Shoots
yeah woops, just wanted to print out what was in it
2018-06-25 23:01:48
Rotonen
the size of things is a bit nontrivial, yeah
2018-06-25 23:03:04
Rotonen
@Shoots actually try memfield with `-Xmx180G -Xms180G`
2018-06-25 23:03:22
Shoots
gets out of memory error
2018-06-25 23:03:36
Shoots
tried that last night
2018-06-25 23:04:00
Rotonen
what's using the memory before you start java? if you have 192, that should still leave 12 free
2018-06-25 23:04:17
Rotonen
and as said, try that with threads=1
2018-06-25 23:04:21
Rotonen
to see if that starts
2018-06-25 23:05:20
Shoots
it shows up as 185gb
2018-06-25 23:05:56
Shoots
if I dont set xmx higher than java uses I get out of mem error
2018-06-25 23:06:48
Rotonen
which field do you try to mine with?
2018-06-25 23:06:57
Rotonen
7 should fit
2018-06-25 23:07:09
Rotonen
8 would not
2018-06-25 23:07:41
Rotonen
try with the old gc from above and both xmx and xms and threads 1
2018-06-25 23:07:57
Rotonen
it almost sounds like your memfielding is double
2018-06-25 23:09:09
Rotonen
also the threads 1 will give me a more exact figure on where the per channel bandwidth lands on those systems
2018-06-25 23:10:19
Rotonen
for ram ballparks when 1min and 5min agree close enough that's a result
2018-06-25 23:17:50
Rotonen
plotted numbers and i pay one eurocent per hour per kilohash on my infra for mining
2018-06-25 23:19:28
Rotonen
no, that's actually not right at all
2018-06-25 23:19:34
Rotonen
yeah, time to sleep :smile:
2018-06-25 23:19:47
Rotonen
(failing way too hard at excel references)
2018-06-25 23:24:17
Shoots
hmmm I had a partial field 8 in my snow/mainnet folder from one time I accidentally had autosnow enabled, I wonder if thats what was causing my issue
2018-06-25 23:24:57
Shoots
when I would launch the miner I noticed it says building field 8, but then never continues to build it.
2018-06-25 23:25:08
Shoots
cause I have autosnow disabled now
2018-06-25 23:27:23
Rotonen
that could explain the 0 hash
2018-06-25 23:27:52
Rotonen
@Shoots actually drop the memfield parametre, drop the xmx, just let it read from the disk and let fscache slowly take over
2018-06-25 23:28:28
Fireduck
Don't forget to fprot the tarball
2018-06-25 23:32:15
Shoots
Is fscache automatic or do I have to enable it?
2018-06-25 23:35:39
Shoots
also if I still have snowfield 6 will it try to load that into mem?