Race Internet - conduit install - be careful

[rsfpostreader]

Just thought I'd post a precautionary warning about conduit install. I'd recommend supervising the HP crews closely to ensure that they actually trench and adequately bury the conduit on your property.

In my case it routes across part of our laneway, which understandably had to be cut, and then through my garden to the house. They didn't really trench -- rather, as someone else on here described, scraped a small ditch, maybe 3" deep, laid the conduit, and then covered it all back up again. I raised the issue and after 3 or 4 attempts finally got the right people from both Race and HP to come out and look at it. They assured me it wasn't a damage risk, so I eventually conceded and accepted the install as I was anxious to just get things up and running after a few months of back and forth delays. I expect we'll have to routinely cover the conduit over time, but am not particularly concerned about it getting otherwise damaged.

What I didn't appreciate then was that when they cut my driveway, they didn't go very deep -- so the conduit is basically not buried at all -- there was so little cover that the asphalt patch has deteriorated in spots and I expect the rest will go away quickly as well. This is what it looks like after ~3 months, with probably 3-4 cars/week up and down the lane (it isn't our primary driveway). While they might (hopefully) be right about the conduit being safe in my garden over the long term, I'm certain it won't be safe on my driveway! (not to mention, it'll look like garbage)

The good news is I am very happy with my connection speed, so Race is doing their part well, it's really HP that is falling flat. I did need to upgrade my Unifi firewall, but that was expected. Wired connections are basically full gig, and wifi is consistent with my access point performance. If you are expecting full gig on your wifi, you will likely be disappointed, and it has nothing to do with Race or your connection.

 

[ptrubey]

Good post. I’d go a step further in the recommendation and I’d recommend hiring your own gardener or contractor to install your conduit yourself. That’s the only way it’ll be done to your own specifications. Digging trenches in our soils is very labor intensive and expensive, so I’m not particularly surprised HP isn’t doing a great job.

That asphalt crossing looks pretty bad. I would not be happy with that myself.

But yes, Race is doing its part quite well. The Internet service has been exemplary. Btw, I ended up ditching my unifi firewall and got a pfSense one. Far better. Let me know if you want more info.

 

[rsfpostreader]

I agree with you on hiring someone directly. I'd seen your post about that, and thought about getting my landscaper to do it. I think we'd both be happier right now if I had -- he was pretty annoyed at HP as is, given they also broke a bunch of sprinkler lines (not surprising) and didn't tell anyone (surprising).

I ran pfSense for years before switching to the USG. I love the power and control on pfSense, but ultimately Unifi's full stack integration with my switches and AP's won out. I still have a pfSense instance running as my "backup plan" in case the USG dies, which has happened before. The USG will only do <80mbits if I have DPI and its other goodies enabled, so I just ordered a their new "Dream Machine Pro" to replace it. If that won't handle gigabit _with_ all the bells and whistles then I'll likely revert to my pfSense.

 

[ptrubey]

Yeah, I tried running DPI and IPS (which is much more CPU intensive) when I had a 30 Mbps connection and the small USG could keep up. DPI info was useless as far as I could tell. Pretty graphs, but so what. And IPS generated a bunch of false positives that I had to track down and realize they were false positives. In the end, proper true corporate grade security is hard to achieve especially in a home environment with all the stupid IOT things that phone home all the time. I mean, our air purifier wants to be on the WiFi. Why? Just so they can get rid of physical switches and fan changes must be done through an app. Sheesh.

So I've pretty much given up having truly high security. The dodgier IOT things I put on the guest WiFi so at least all they can talk to is the router. I make sure I have good Windows firewall settings, etc.

The reason I ended up getting rid of the USG is that it wasn't 100% rock solid. It would freeze every month or so. And it ran very hot. I opted not to go with the Dream Machine when a cursory glance at the forums showed lots of bugs.

I'm loving my pfSense firewall. Two big differences. One is the much more detailed and useful logging it does. I found a few network issues that way on my network just by looking at the logs. Second, it has a DNS resolver. So instead of sending DNS queries out to Race's or Google's DNS server, the firewall resolves (and caches) all DNS queries. This actually makes a noticeable difference. Complex web pages snap up much quicker now than they used to. While Race and Google's DNS servers are good, they are still many milliseconds away, and have significant processing loads meaning queries take a lot longer than the pfSense one.

I don't miss the lack of Unifi integration with the firewall. The one thing I like to have is a real time graph of network usage. It's amazing the info you get from that constantly updated graph. For that, I've always done that through the SolarWinds SNMP monitor (the free one), that queries the firewall every three seconds, so that I can see the last 30 minutes of activity at all times. Here's what this quiet Saturday morning looks like:

 

[RSF Networks]

We had a similar experience. HP buried the conduit within an inch of the surface. Even worse they brought it 35 feet away from the designated location outside the wrong room. We corrected it for ourselves. Likewise, to avoid any delays in installation.

John

 

[MadPersianPrince]

I got the conduit install done back in late December or early Jan and also noticed the not so great conduit install. Also, about 1-2 feet of the fiber cable is exposed to outdoor elements before it punches through into my house. Even if the cable is outdoor rated it, the insulation can still degrade after a couple of years. It works (for now) so, I guess I'll see. I am also very happy with the speeds (getting around 940/940 on wired network). I wish the price for a static IP wasn't so high (but nothing ddns cant fix). My house is wired with CAT6A w/ Cisco RV345 router and 16 port gigabit switch with 8 PoE ports. I have had good luck with Unifi-AC-Pro access points (I don't use their cloud software, just managed through my PC), of course I am not getting the same speeds (getting around 500/500) as the wired network due to WNIC limitations on the end device (+ many other limitation factors that are come with WiFi). To be frank, I never see Access points / routers provide WiFi speeds as advertised.

 

[ptrubey]

I got the conduit install done back in late December or early Jan and also noticed the not so great conduit install. Also, about 1-2 feet of the fiber cable is exposed to outdoor elements before it punches through into my house. Even if the cable is outdoor rated it, the insulation can still degrade after a couple of years. It works (for now) so, I guess I'll see. I am also very happy with the speeds (getting around 940/940 on wired network). I wish the price for a static IP wasn't so high (but nothing ddns cant fix). My house is wired with CAT6A w/ Cisco RV345 router and 16 port gigabit switch with 8 PoE ports. I have had good luck with Unifi-AC-Pro access points (I don't use their cloud software, just managed through my PC), of course I am not getting the same speeds (getting around 500/500) as the wired network due to WNIC limitations on the end device (+ many other limitation factors that are come with WiFi). To be frank, I never see Access points / routers provide WiFi speeds as advertised.

To protect the fiber cable from UV light (it is actually waterproof, but the sun could be worse over time) wrap it with this.

https://www.amazon.com/Alex-Tech-25ft-Tubing-Conduit/dp/B07TCDTFL2

Sounds like you have a good setup.

 

[RSF Networks]

I got the conduit install done back in late December or early Jan and also noticed the not so great conduit install. Also, about 1-2 feet of the fiber cable is exposed to outdoor elements before it punches through into my house.

The type of fiber they use is fine for outdoors. It is designed for it. It even has a messenger wire so it could be installed aerially. I'm not sure aerially is actually a word but I mean to say, you could run it from a telephone pole to the house in a different town that isn't doing underground.

My concern with what you've mentioned would be small animals gnawing on it. Post a photo if you can.

Additionally, Race has specs published for this and what you've described doesn't meet them. Here's a link to Race's specs:

Link to Race Install Specs PDF

Personally, I never leave fiber exposed unless it's armored type. For the indoor portion, it should be wrapped in InnerDuct which is a brand name for what is technically known as corrugated loom tubing.

Link to InnerDuct Products

We often use the Panduit version of this product.

For the outdoor section, it really needs to be in gray electrical schedule 40 PVC conduit and penetrate the house using an LB which looks like this:


I would recommend calling Race and asking them to come out and remediate the issue.

 

[RSF Networks]

My house is wired with CAT6A w/ Cisco RV345 router and 16 port gigabit switch with 8 PoE ports. I have had good luck with Unifi-AC-Pro access points (I don't use their cloud software, just managed through my PC), of course I am not getting the same speeds (getting around 500/500) as the wired network due to WNIC limitations on the end device (+ many other limitation factors that are come with WiFi). To be frank, I never see Access points / routers provide WiFi speeds as advertised.

Wow. Cat 6A is serious business. Hard to pass certification tests. Very strict standard. It's interesting you had that done. MultiGig is here. It's nice to be ready to take advantage of it. Even with MultiGig capable switches and APs, you still need an excellent client device to get near a gigabit speed on any one particular client. High end MacBook Pros are the only laptops currently available that can do 3 streams which can connect at 867 MBps although the max theoretical is 1300 MBps.

Since you mentioned Unifi, we run a cloud controller for our clients. Nice monitoring features. Here's a screenshots of the dashboard:

 

[MadPersianPrince]

@ptrubey: Thank you. Those looms are good but I made a major miscalculation on the amount of fiber exposed when I went out to take a picture of it today. There is about 10 feet of fiber exposed. I can use the loom for the gap from the orange conduit to the pull box but the other area should be in a conduit. This area doesn't get much sunlight so I wont really have to worry about deterioration from UV but like @RSF Networks mentioned, it has potential to be damaged by pests.
See picture below (disregard the conduit with an X on it, that is another issue I have to follow up with a different contractor ?)
@RSF Netowkrs: Thanks for the info in the Race specs; I'll follow up with them to see if they can remedy.

 

[MadPersianPrince]

Wow. Cat 6A is serious business. Hard to pass certification tests. Very strict standard. It's interesting you had that done. MultiGig is here. It's nice to be ready to take advantage of it. Even with MultiGig capable switches and APs, you still need an excellent client device to get near a gigabit speed on any one particular client. High end MacBook Pros are the only laptops currently available that can do 3 streams which can connect at 867 MBps although the max theoretical is 1300 MBps.

Since you mentioned Unifi, we run a cloud controller for our clients. Nice monitoring features. Here's a screenshots of the dashboard:

View attachment 43

True, I heard about the strict CAT6A standards. I didn't get them certified because I don't have proper the tool for that. I terminated the connections myself and tested with a LRAT-1000 LinkRunner for all that is worth. I am not a data cabling guy but I am in the Network Engineering field for all that is worth too ?. I guess I'll find out if it passes once I go to 10Gig speeds!

Nice, I can see how the Unifi Cloud is a very useful tool if you manage multiple clients on it.

 

[ptrubey]

While Race does not offer it yet, they are set up for 10G up and down using their Calix equipment, and certainly the fiber network in the ground can support it. Subscribers would need to have 10G switches and network cards. As it is, the few times I've had to download or upload huge files, the gigabit link has been mind blowing. Soooo quick. I guess in a couple of years we'll get used to it and want even faster links

 

[RSF Networks]

True, I heard about the strict CAT6A standards. I didn't get them certified because I don't have proper the tool for that. I terminated the connections myself and tested with a LRAT-1000 LinkRunner for all that is worth. I am not a data cabling guy but I am in the Network Engineering field for all that is worth too ?. I guess I'll find out if it passes once I go to 10Gig speeds!

Nice, I can see how the Unifi Cloud is a very useful tool if you manage multiple clients on it.

We did a campus style hotel with 23 buildings last year. 600 runs of CAT 6A. It was problematic for a number of reasons. We certified with Fluke DTX-1800. We had issues with cheap keystones from Monoprice. Had to do the first building twice. The most important tip I learned is you really have to maintain the twists all the way. You can't have the final 1/4" untwisted or it will fail on NEXT (Near End CrossTalk).

 

[RSF Networks]

While Race does not offer it yet, they are set up for 10G up and down using their Calix equipment, and certainly the fiber network in the ground can support it. Subscribers would need to have 10G switches and network cards. As it is, the few times I've had to download or upload huge files, the gigabit link has been mind blowing. Soooo quick. I guess in a couple of years we'll get used to it and want even faster links

I'm interested in what's going on with this. It's a GPON delivery to the homes I've seen. The G stands for gigabit. I've read there's newer 10 gig stuff. I would like to know what exactly Race has in place. Is there any info on that? When I drove past the hut I had immediate concerns just based on the type of air conditioning. Not data center grade. I also wonder what the upstream bandwidth availability is. I assume they are co-located with AT&T or Verizon somewhere nearby but the hops I've seen only show LA. I'd love to know more.

Article from Adtran 10G PON

 

[RSF Networks]

@RSF Networks : Thanks for the info in the Race specs; I'll follow up with them to see if they can remedy.
View attachment 44

Wow. Crime scene. Just wow.

 

[ptrubey]

I'm interested in what's going on with this. It's a GPON delivery to the homes I've seen. The G stands for gigabit. I've read there's newer 10 gig stuff. I would like to know what exactly Race has in place. Is there any info on that? When I drove past the hut I had immediate concerns just based on the type of air conditioning. Not data center grade. I also wonder what the upstream bandwidth availability is. I assume they are co-located with AT&T or Verizon somewhere nearby but the hops I've seen only show LA. I'd love to know more.

Article from Adtran 10G PON

They use standard GPON currently. So that's shared 2.5 Gbps downlink, and shared 1.2 Gbps uplink. I believe (but don't know) that they use 1-32 splitters, but at some point a Race engineer told me they try to keep it at a max of 1:24 split and they will balance out the shared links if they see multiple high capacity users on one PON. I keep a bandwidth monitor on all the time on my second monitor and the number of times I spike to 800 Mbps or so is very rare indeed, so sharing 2.5 Gbps bandwidth among 24 neighbors is likely to not be noticeable.

The central office has two diversely physically routed 10 Gbps links to the Internet. From what I can tell, they do not have a lot of routers. They seem to have leased 10 Gbps links directly to an ISP hotel in Los Angeles where all the major players put their content delivery networks. For example, here's a traceroute I just did to a Youtube playback server:



As you can see, it goes from my home router, to the Race router in the RSF central office, and then to a Race router in Los Angeles, and then to a google (Youtube) router. And as you can see, this results in a 4ms round trip times to/from Youtube. I wrote about this pretty extensively at https://rsfpost.com/2020/01/30/rsf-connect-review/

As far as air conditioning goes, they have two (or was it three?) independent units so I wouldn't worry too much about that. And no, no ISP co-location, they are their own ISP and go directly to the source (massively connected ISP hotels) for connectivity.

 

[RSF Networks]

Just an observation. My point on the air conditioners is, if they took a shortcut on that, they likely took others. I'm expecting Liebert for a datacenter. Not saying it won't work, just saying, it's done on the cheap. Maybe other aspects might be as well.

On the oversubscription rate, how many subscribers are we expecting? 1,500? Seems like 50:1 would be a good ratio. I'm mildly surprised to admit, it seems like 20 gigs at the headend should be okay for a while.

I have to plead ignorance on this but what are ISP hotels? I'm not familiar with that term. You mean like a PoP (point of presence) for co-location?

I looked it up and they're using CoreSite for their co-location partner to get them to the peering point. I'm a bit in the dark about how they get from RSF to LAX in one hop.

 

[ptrubey]

Just an observation. My point on the air conditioners is, if they took a shortcut on that, they likely took others. I'm expecting Liebert for a datacenter. Not saying it won't work, just saying, it's done on the cheap. Maybe other aspects might be as well.

The building itself, electrical, AC and power backup (generator) was all designed, built and owned by the RSF Association. And while many aspects of this infrastructure was designed by outside engineers and then by committee decisions(!), the AC system itself was probably most influenced by ... me. I was on the tech committee when those kinds of decisions were being made. I personally think that a data center grade AC system is overkill and would be significantly louder than the inverter type system they have now (and that matters in a residential neighborhood). An inverter system is much more robust and reliable than your typical residential AC compressor. The AC is monitored (I HOPE!) and if one goes offline, they have one or two backups. The reality is that there isn't much active equipment in that room. There aren't any heavy duty servers that pump out heat. All you have is an AC to DC inverter for the telecom grade equipment power supply, some rack spaces for the Calix GPON frame and cards, a router, a 10 Gbps switch, UPS batteries, and maybe a lightly loaded monitoring server or two. If worse came to worse, opening the door and sticking a fan in it while the AC got worked on would work.

On the oversubscription rate, how many subscribers are we expecting? 1,500? Seems like 50:1 would be a good ratio. I'm mildly surprised to admit, it seems like 20 gigs at the headend should be okay for a while.

Yes, I would think 1,500 would be near an upper limit many years from now. So yes, you are correct, the 20 Gbps is overkill, especially now, and also especially also when you consider that our demographics trend quite a bit older than the general population, which would likely mean even less overall usage. That is one of the nice things about owning our own network and having some say in its operation. On the tech committee, some of us were very leery of handing it over to a Cox or Spectrum since they would likely overload the uplinks like they usually do. That part of the network has worked out exactly the way I hoped it would.

I have to plead ignorance on this but what are ISP hotels? I'm not familiar with that term. You mean like a PoP (point of presence) for co-location?

Yes. But more than an old school POP since these days the major content providers put CDN servers at these locations as well. So Netflix, Amazon, Google, etc. will have essentially a caching server in Los Angeles, and a few on the east coast. As you can see from the traceroute, we are right "next door" to google. With that kind of latency and that kind of bandwidth (I got 800 Mbps upload to Youtube last week), if the Youtube server were on my local LAN, I wouldn't be able to tell the difference without measuring equipment.

I looked it up and they're using CoreSite for their co-location partner to get them to the peering point. I'm a bit in the dark about how they get from RSF to LAX in one hop.

It's the modern telecom network, which underlies all data comm. Each long haul fiber strand is now carrying anywhere from 100 Gbps to 800 Gbps point to point. And these strands can be bonded to even higher bit rates. Basically, Race is leasing 10 Gbps data transfer from the base of Encinitas Blvd (where it crosses over the bridge into RSF) to their Los Angeles location. And there is another 10 Gbps link further south. That 10 Gbps gets multiplexed along with lots of other dedicated bandwidth customers onto a 100 Gbps or 400 Gbps or whatever long haul fiber network. Telecom multiplexing is all deterministic (as opposed to the stochastic queue and forward nature of packet networks), so it doesn't matter how many mux and demux steps (within reason) Race's dedicated 10 Gbps link does on and off different fiber strands. In the end they have what looks like 10 Gbps clear from the RSF central office to their Los Angeles end point.

So to answer your question, they don't use routers, they (and it isn't Race, it is a long haul fiber provider) use lower level telecom equipment that combine dedicated bandwidth data streams onto available long haul fiber. So undergirding the router network we can see using ping and traceroute is an invisible (to us) proprietary telecom network that does the real heavy lifting.

This all ties into the evolution of the Internet which has changed dramatically from the early days of distributed servers to today where everyone is going to the same top ten locations: Youtube, Amazon, AWS (part of Amazon, and it runs half of the Internet servers), Google, various smaller cloud providers, etc.

 

[RSF Networks]

Good info, Phil. I'm out of touch with the carrier stuff. I was a telecom buyer back in the late 90's through 2007 but I'm out of step with the goings on of the past decade or so. My first major rollout was 50 sites of private IP over frame relay with UUNet. Only later to replaced with DS-3s from Worldcom. I feel old just thinking about it.

 

[ptrubey]

Good info, Phil. I'm out of touch with the carrier stuff. I was a telecom buyer back in the late 90's through 2007 but I'm out of step with the goings on of the past decade or so. My first major rollout was 50 sites of private IP over frame relay with UUNet. Only later to replaced with DS-3s from Worldcom. I feel old just thinking about it.

I know what you mean. I have a book in my library that shows the major Internet links in use circa 1995. It has changed a teeny weeny bit since then.