2018-09-12 00:14:36
@Fireduck but is it web scale? :stuck_out_tongue:
Rotonen
2018-09-12 00:16:49
Just a thought - maybe we can re-use the smaller snowfields? So Snowfield 1 would be 1/2 snowfield 0 + 1/2 generated snowfield 1, field 2 1/2 field 1 and so on
mjay
2018-09-12 00:17:13
This way after a snowstorm you need to download only half as much
mjay
2018-09-12 00:17:35
and generation can use 2 threads :smile:
mjay
2018-09-12 00:18:02
would need a hardfork however
mjay
2018-09-12 00:19:02
I thought of that and rejected it. I didn't want to weaken the field generation in any way. A lot of the pow security depends on it being hard to reproduce in part
Fireduck
2018-09-12 00:20:48
So I installed nfs server and clients and Arktika still comes up with connection refusal errors, iterating though each node:
aikida3k
2018-09-12 00:21:02
``INFO: Subscribed to work
Sep 11, 2018 7:11:46 PM snowblossom.miner.LayerWorkThread run
WARNING: Error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:221)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:202)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:131)
at snowblossom.mining.proto.SharedMiningServiceGrpc$SharedMiningServiceBlockingStub.getWords(SharedMiningServiceGrpc.java:163)
at snowblossom.miner.FieldSourceRemote.readWordsBulk(FieldSourceRemote.java:100)
at snowblossom.miner.BatchLayerWorkThread.runPass(BatchLayerWorkThread.java:80)
at snowblossom.miner.LayerWorkThread.run(LayerWorkThread.java:189)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.1.122:2311
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:635)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
... 11 more
aikida3k
2018-09-12 00:24:11
I see it is refusing connection on :2311 but I don't have a firewall up; ufw is inactive
aikida3k
2018-09-12 00:24:25
Sorry if I am butting in on a converstation
aikida3k
2018-09-12 00:24:55
Not at all
Fireduck
2018-09-12 00:25:26
Well, it is absolutely getting a connection refused
Fireduck
2018-09-12 00:25:51
Maybe check lsof on the nodes and make sure the listen is open as expected
Fireduck
2018-09-12 00:26:11
are the nodes listening on localhost?
Rotonen
2018-09-12 00:37:23
i don't exactly know what I am supposed to be seeing with lsof -u arktika
aikida3k
2018-09-12 00:45:08
@Rotonen I'm not quite sure how to find that out
aikida3k
2018-09-12 00:46:56
the loopback resolves to the hostname in /etc/hosts but how do I check if the nodes are listening on localhost
aikida3k
2018-09-12 01:10:32
@Fireduck what version of GRPC is used?
aikida3k
2018-09-12 01:25:09
@aikida3k I believe arktika connects to a pool, is there a firewall blocking it?
Clueless
2018-09-12 01:25:24
firewalls usually just drop packets, but can refuse connections as well
Clueless
2018-09-12 01:25:42
Has anyone tried tools like this?
https://github.com/prsyahmi/GpuRamDrive RamDrive that is backed by GPU Memory
mjay
2018-09-12 01:26:38
@Clueless no firewall enabled ufw inactive
aikida3k
2018-09-12 01:26:47
@mjay while it's possible to store the snow in gpu memory, most gpus only have 4-8gb, and are expensive as hell. So you'd still be bottlenecked severely by disk/ram io
Clueless
2018-09-12 01:27:23
The idea was to use the memory while the GPUs are mining something else thats not using ram
mjay
2018-09-12 01:27:55
@aikida3k `lsof -i :23380`
Clueless
2018-09-12 01:29:15
make sure your pool is listening and connectable
Clueless
2018-09-12 01:31:30
@aikida3k I don't use a port when connecting to the Arktika server I just point the miners to the server ip. Running Ubuntu Server 18.04.1 LTS
SidGrip
2018-09-12 01:32:06
Yeah, the port isn't configurable
Fireduck
2018-09-12 01:33:17
Not sure which grpc version.
Fireduck
2018-09-12 01:33:31
Not at a computer
Fireduck
2018-09-12 01:33:36
@aikida3k
1. Basically, make sure the pool is actually listening `lsof -i :23380`
2. Make sure you can connect to it from the pool server `telnet localhost 23380`
3. Make sure you can connect to it from the miner server `telnet pool_ip 23380`
Clueless
2018-09-12 01:33:43
GRPC would not give connection refused
Clueless
2018-09-12 01:33:58
it would connect first, then fail some other way.
Clueless
2018-09-12 01:52:05
ok telnet localhost 23380 fails and telnet pool_ip 23380 times out but losf -i says the pool is listening
aikida3k
2018-09-12 01:57:05
@SidGrip That's what the config files are doing, correct? Telling arktika which layer should have which chunks and pointing to the file server to get the chunks. I haven't specified a port to use in my config files, I wrote mine similar to the example
aikida3k
2018-09-12 01:57:36
Correct, no ports
Fireduck
2018-09-12 01:57:51
It uses a port but it isn't configurable
Fireduck
2018-09-12 01:58:24
Someone said PoolMiner, you aren't running PoolMiner right?
Fireduck
2018-09-12 01:58:34
Needs to be arktika everywhere
Fireduck
2018-09-12 02:00:46
Yes, just double check your layer_count= the number of layers your using and that your layers and numbered correctly, took a few tries to figure it out :stuck_out_tongue_closed_eyes:
SidGrip
2018-09-12 02:03:46
Might just be a matter of waiting
Fireduck
2018-09-12 02:04:10
An artika that is loading into memory won't open the port until it is done loading
Fireduck
2018-09-12 02:04:13
I think
Fireduck
2018-09-12 02:04:25
i have pool miner running right now on the file server but I've tried shutting down the pool miner and just running arktika on all nodes
aikida3k
2018-09-12 02:04:54
maybe that makes sense
aikida3k
2018-09-12 02:06:50
any ideas why telnet localhost 23380 fails, or it will fail until its loaded, then the port should open and the telnet would work
aikida3k
2018-09-12 02:07:18
maybe you have a firewall rule in place that blocks access?
mjay
2018-09-12 02:07:33
ufw is inactive
aikida3k
2018-09-12 02:08:19
iptables -L -n
mjay
2018-09-12 02:08:21
empty?
mjay
2018-09-12 02:08:23
The pool miner is needed, can't help with telnet since I did not have to use it
SidGrip
2018-09-12 02:08:51
Needed for what?
Fireduck
2018-09-12 02:09:54
Lol, miss read thought refering to pool miner as Arktika server
SidGrip
2018-09-12 02:11:05
@mjay yep empty
aikida3k
2018-09-12 02:11:23
okay no firewall issue then
mjay
2018-09-12 02:11:35
This is my setup setup so far, haven't had time to get back to add more miners yet
SidGrip
2018-09-12 02:11:36
Arktika server
mine_to_address=
pool_host=snowblossom.hamster.science
layer_count=2
layer_0_type=mem
layer_0_range=0,127
layer_0_threads=100
layer_1_type=file
layer_1_path=//media/sidgrip/NVME/snow_chunked
layer_1_threads=2
selected_field=7
Node Miner
mine_to_address=
pool_host=snowblossom.hamster.science
layer_count=3
layer_0_type=remote
layer_0_range=0,42
layer_0_threads=100
layer_0_host=192.168.2.155
layer_1_type=mem
layer_1_range=43,127
layer_1_threads=100
layer_2_type=file
layer_2_path=//media/sidgrip/2TB/Torrents
layer_2_threads=2
selected_field=7
SidGrip
2018-09-12 02:18:34
I'll try letting it run overnight and see how it is in the morning and go from there.
aikida3k
2018-09-12 02:19:15
It will say if it is loading blocks
Fireduck
2018-09-12 02:19:46
I know I have to reallocate some of the chunks because I got Heap overflow errors when I allocated 6GB chunks to 8GB RAM
aikida3k
2018-09-12 02:20:50
I found that allocating half of your total ram works best for me
SidGrip
2018-09-12 02:21:58
Right now it get to the point of subscibing to work and then the io.grpc.StatusRuntimeException:UNAVAILABLE error shows
aikida3k
2018-09-12 02:22:33
Do all of the nodes do that?
Fireduck
2018-09-12 02:22:40
yes
aikida3k
2018-09-12 02:32:27
At intervals it shows it is trying to work-- I guess connecting with the pool and then it reverts back to the errors
aikida3k
2018-09-12 02:33:37
Caused by: java.net.ConnectException: Connection refused
... 11 more
Sep 11, 2018 9:29:41 PM snowblossom.miner.Arktika printStats
INFO: 15 Second mining rate: 0.000/sec - at this rate ∞ minutes per share (diff 22.000)
Sep 11, 2018 9:29:41 PM snowblossom.miner.Arktika printStats
INFO: 1-min: 0.000/s, 5-min: 0.000/s, hour: 0.000/s
Sep 11, 2018 9:29:41 PM snowblossom.miner.Arktika printStats
INFO: we seem to be stalled, reconnecting to node
Sep 11, 2018 9:29:41 PM snowblossom.miner.Arktika$WorkUnitEater onError
INFO: Error talking to mining pool: io.grpc.StatusRuntimeException: UNAVAILABLE: Channel shutdownNow invoked
Sep 11, 2018 9:29:41 PM snowblossom.miner.Arktika subscribe
INFO: Subscribed to work
Sep 11, 2018 9:29:41 PM snowblossom.miner.Arktika printStats
INFO: Queues: {0,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10001,10000,10000,10001,10000,0,}
Sep 11, 2018 9:29:41 PM snowblossom.miner.Arktika printStats
INFO: Shares: 0 (rejected 0) (blocks 0)
Sep 11, 2018 9:29:43 PM snowblossom.miner.LayerWorkThread run
WARNING: Error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:221)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:202)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:131)
at snowblossom.mining.proto.SharedMiningServiceGrpc$SharedMiningServiceBlockingStub.getWords(SharedMiningServiceGrpc.java:163)
at snowblossom.miner.FieldSourceRemote.readWordsBulk(FieldSourceRemote.java:100)
at snowblossom.miner.BatchLayerWorkThread.runPass(BatchLayerWorkThread.java:80)
at snowblossom.miner.LayerWorkThread.run(LayerWorkThread.java:189)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.1.130:2311
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:635)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
... 11 more
aikida3k
2018-09-12 02:34:52
So there is a problem talking to the pool: INFO: Error talking to mining pool: io.grpc.StatusRuntimeException: UNAVAILABLE: Channel shutdownNow invoked
aikida3k
2018-09-12 02:35:58
I guess that could be because the port is still closed. So i'll try to fix the field allocation and let it run over night
aikida3k
2018-09-12 02:36:15
what mining pool are you pointing the Arktika nodes to
SidGrip
2018-09-12 02:36:25
vauxhall
aikida3k
2018-09-12 02:38:11
Have you tried to connect locally to eachother without the port#
SidGrip
2018-09-12 02:41:01
I can telnet to port 22, I can transfer files
aikida3k
2018-09-12 02:43:01
as far as telnet ip : conncetion refused
aikida3k
2018-09-12 02:43:16
Since I don't know your setup, I really don't know sir, Every runs fine locally on linux & simple to setup
SidGrip
2018-09-12 02:44:38
i'm running ubuntu 16.04 on dual cpu servers. the pool miner works great but the issue is with getting arktika going as it was similar to the example
aikida3k
2018-09-12 02:44:53
anyways i'll keep trying
aikida3k
2018-09-12 03:47:03
feel free to open an issue on github and share the exact configs
Fireduck
2018-09-12 03:47:06
I'll get into it
Fireduck
2018-09-12 04:43:05
ok, I need a name
Fireduck
2018-09-12 04:43:18
the hardest part of any project
Fireduck
2018-09-12 04:43:51
JustSomeNameIWillRemember
mjay
2018-09-12 04:44:03
oh
Fireduck
2018-09-12 04:44:53
Is there any specific reason why well44497b was chosen?
mjay
2018-09-12 04:45:12
haha, I've been waiting for that question
Fireduck
2018-09-12 04:45:14
you get a prize
Fireduck
2018-09-12 04:45:22
:smile:
mjay
2018-09-12 04:45:28
its the huge internal state?
mjay
2018-09-12 04:45:29
It is a large state prng
Fireduck
2018-09-12 04:45:44
yeah, larger state means harder to checkpoint the state of the snowfall
Fireduck
2018-09-12 04:45:52
makes sense
mjay
2018-09-12 04:46:01
which makes it harder to build sections as needed in a GPU or ASIC for example
Fireduck
2018-09-12 04:48:19
but that wasn't enough state so I added the snow monster anyways
Fireduck
2018-09-12 04:48:29
which has a 256mb internal state, effectively
Fireduck
2018-09-12 04:48:40
so probably not important anymore, but whatever
Fireduck
2018-09-12 04:49:04
currently crawling through the code. You put a lot of thought into it
mjay
2018-09-12 04:49:41
you might add checkpoints to the initial write, and thats it as far as I can see
mjay
2018-09-12 04:50:24
sure, the initial write isn't a big deal
Fireduck
2018-09-12 04:50:42
Maybe mark the pages that have not been touched since then in a bitmap
mjay
2018-09-12 04:50:55
but thats just a few if I´m correct
mjay
2018-09-12 04:51:03
probably
Fireduck
2018-09-12 04:51:48
less than 2%
mjay
2018-09-12 04:53:04
yeah, I'm having a hard time with that math
Fireduck
2018-09-12 05:11:16
you could have travelled along a multiplicative group mod n to reach all pages
mjay
2018-09-12 05:12:41
but not sure if this would open up any issues
mjay
2018-09-12 06:23:06
oh god, that sounds like group theory math or something
Fireduck
2018-09-12 06:23:10
which I know nothing about
Fireduck
2018-09-12 07:04:20
Its not that complicated. Pageindex for round N would be (N * (any prime) + 1) % (total elements)
mjay
2018-09-12 07:10:23
Do you know of any C/C++ implementation of well44497b?
mjay
2018-09-12 07:10:37
I found this one: http://www.iro.umontreal.ca/~panneton/well/WELL44497a.c
But gives different results
mjay
2018-09-12 07:12:13
converting the java code is a mess, bit-shifts + big/little endian. Thats no fun
mjay
2018-09-12 08:51:06
find a microsoft integrator to help with that, that’s the kind of bullshit they deal with daily
Rotonen
2018-09-12 08:51:40
just look at the struct which makes up an AD SID
Rotonen
2018-09-12 13:15:06
@mjay don't want anything that predicable, the next word to use is determined as part of the prng state. You can only determine the path by doing the work which means having the whole field
Fireduck
maybe someone can help me
ork block load error: java.lang.RuntimeException: Unable to select a field of at least 7. Availible: []
2018-09-12 18:21:57
wrong path for snow file? snowfile incomplete?
fydel
nvm
im just stupid
2018-09-12 18:22:24
we can't help you with stupid, but you are in good company
Fireduck
2018-09-12 18:22:47
confirmed.
fydel
Unexpected length on snowblossom.7.deck.a
ok maybe im not that stupid
2018-09-12 18:26:18
probably not completely downloaded
Fireduck
it is
2018-09-12 18:26:28
what length is that file?
Fireduck
2018-09-12 18:27:16
total 134349260
drwxr-xr-x 2 nerd nerd 4096 Aug 19 19:59 .
drwxr-xr-x 9 nerd root 4096 Aug 19 22:56 ..
-rw-r--r-- 1 nerd nerd 134217728 Aug 19 22:09 snowblossom.7.deck.a
-rw-r--r-- 1 nerd nerd 131072 Aug 19 19:59 snowblossom.7.deck.b
-rw-r--r-- 1 nerd nerd 128 Aug 19 19:59 snowblossom.7.deck.c
-rw-r--r-- 1 nerd nerd 137438953472 Aug 19 23:00 snowblossom.7.snow
Fireduck
t 128M Jun 21 01:49 snowblossom.7.deck.a
how can i make it show in
ok
wait
where can i direct download this
2018-09-12 18:28:44
you can't, hosting files this size isn't cheap so no one does it
Fireduck
2018-09-12 18:28:50
bittorrent is open option
Fireduck
2018-09-12 18:30:28
^^ if you are in europe.
fydel
its deck a
and im using that
which seems to be broken
but its weird cause i used those links before
oh problem is snow itself
not sure why
2018-09-12 18:36:37
i am mining on that file right now. seems to work here.
fydel
y the snow seemed to cache only 71gb
after wget
2018-09-12 18:38:30
try aria2. it is faster and fancier.
fydel
its not faster
face it
2018-09-12 18:39:23
if you use multiple connections.
fydel
2018-09-12 18:41:30
I use deluged and deluge-console
Fireduck
2018-09-12 18:41:36
seem to work well for me
Fireduck
this is so odd
ubuntu seems to change the file size
of snowblossom
snowfield*
2018-09-12 19:40:18
@Fireduck @fydel
I was talking to some huge torrent seeders (50,000+ torrents seeding)
They seem to advocate for transmission.
Clueless
2018-09-12 19:40:26
Seems deluge bogs down after 2,000 torrents
Clueless
2018-09-12 19:40:39
that's a nutso community
Clueless
is it usual some time till high velocity kicks in?
2018-09-12 19:42:11
ah.nice. i am am seeding with transmission.
fydel
2018-09-12 19:42:57
@dk3 what do yo umean? it takes a little bit of time to load the full snowfield into ram
Clueless
but it shows loading?
i noticed when pre caching it shows
but if you use memfield true only
it does not
2018-09-12 19:45:23
@dk3 if on linux, do `free -h` several times, and see if your cache is diminishing
Clueless
2018-09-12 19:45:36
that would be the snowfield being cached into ram
Clueless
it started mining
but stopped
how can i clear cache
i was already using that command
its supposed to appear as used
and not as cached?
total used free shared buffers cached
Mem: 251G 251G 556M 872K 2.0M 41G
take a look
and i have big drops
2018-09-12 19:51:11
@dk3 depends, memfield=True I think would be used
Clueless
2018-09-12 19:51:20
if you're waiting on OS to cache the file, it'd be cached
Clueless
it just dissappears from cached
and goes into used
how can i reserve
2018-09-12 19:58:06
yeah, that's fine
Clueless
https://wiki.openjdk.java.net/display/shenandoah/Main Shenandoah GC Shenandoah is an ultra-low pause time garbage collector that reduces GC pause times by performing more garbage collection work concurrently with the running Java program. CMS and G1 both perform concurrent marking of live objects. Shenandoah
guys
couldnt this be useful for us?
2018-09-12 22:36:57
dunno, try
Rotonen
2018-09-12 22:37:16
i'm sure a pull request with a repeatable benchmark will get evaluated and merged
Rotonen
found a "flaw" :open_mouth:
getting 30mh
right now
2018-09-12 22:41:26
what flaw?
mjay
its not just a flag
you would have to compile the source again
2018-09-12 22:42:50
this is no real hashrate, is it?
mjay
it is
found 1 block so far
2018-09-12 22:43:59
so .. what hardware are you using?
mjay
its not about the hardware
2018-09-12 22:46:21
1mh/s using pen & paper - easy
mjay
2018-09-12 22:46:40
whatever it is you're doing currently is probably not useful, but if you want us to listen, set a block remark
Rotonen
remark next snowfield in some hours :wink:
2018-09-12 22:48:34
only since you push it - i'll call you a liar in 3 hours, then, tick tock
Rotonen
dont wonder it was just a silly joke
but i ve found doing "sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
"
from time to time
might boost your hashrate
or even just once while mining
2018-09-12 22:52:48
it reduces memory fragmentation as it clears the caches
mjay
2018-09-12 22:53:09
218c218
< http://logger.info(String.format("15 Second mining rate: %s/sec %s", df.format(rate), block_time_report));
---
> http://logger.info(String.format("15 Second mining rate: %s/sec %s", df.format(rate * 10), block_time_report));
10x speed patch! :thinking_face:
mjay
nice pull
anyway to set static diff
on pool?
2018-09-12 22:54:18
you need to hack the source
mjay
2018-09-12 22:54:25
yea, possible
mjay
2018-09-12 22:54:34
someone should bother to write an actual PID for the miner to adjust the diff
Rotonen
2018-09-12 22:55:13
and also using that for autotuning the thread count would help
Rotonen
2018-09-12 22:58:23
@dk3 difficulty is set in MiningPoolServiceAgent.java, look for working_diff
mjay
2018-09-12 22:58:58
https://github.com/snowblossomcoin/snowblossom/blob/058e11c75f8dbb56844486d133bd0e82802e8e13/miner/src/MiningPoolServiceAgent.java#L169 ```
WorkInfo wi = new WorkInfo(wu, info, blk, info.working_diff);
```
Rotonen
2018-09-12 22:59:29
it should actually not be that hard to just have a config to override that, but the current 'autotune' is not *that* bad
Rotonen
2018-09-12 22:59:38
@dk3 what's the issue you're solving, actually?
Rotonen
2018-09-12 23:00:02
i'm getting a vibe there's something to game with low diff shares on the pool implementation
Rotonen
2018-09-12 23:47:40
I would be surprised but not shocked
Fireduck