The following transcript has been edited for length and readability. Listen to the entire discussion here on The Broadband Bunch. The Broadband Bunch is sponsored by ETI Software.
Brad Hine:
Hello, everyone in broadband land. Welcome to another episode of The Broadband Bunch. I’m your host, Brad Hine, bringing you stories, stats, and samples from the world of broadband. Our guest today has spent over 20 years in the broadband industry and currently is focused on teaching and enabling how service providers deliver the most optimized internet services in the toughest areas of the United States. The number one goal is connecting businesses and communities better and faster by creating better solutions and architecture around things like Middle Mile and IP transit and providing knowledge of how rural ISPs can achieve the most efficient practices to run their community services and achieve a hundred percent end-to-end uptime. He is the president and founder of Capcon Networks based in Austin, Texas. Please welcome Offir Schwartz. Welcome to the bunch, Offir.
Offir Schwartz:
Thanks for having me. What a great intro. Thank you.
Brad Hine:
We in Dallas, or you became aware of the Broadband Bunch in Dallas at Connected America. I wanted to give them a shout-out. Thanks for all that. So you attended that show in Dallas. Tell me a little bit about what you thought of connected America.
Offir Schwartz:
I really enjoyed it. Um, they had a unique format for setting up meetings at the show through their app, and it was exceptional at setting up meetings. I have to say that it’s probably one of the best formats for networking that I’ve ever experienced in a trade show. And that was a very small trade show. And so I was really impressed with it. I thought it was fantastic.
Brad Hine:
Yeah. I agree with you. I think it was very easy to engage everyone there, and it was easy to find everyone too. We were all in the same building over a few floors. Tell us a little bit about Capcom Networks and how it got started and, and why, and then I’ll jump into some of the drivers for you getting into the industry.
Offir Schwartz:
Okay. Yeah, so why it got started and how it got started is probably a typical and familiar story for a lot of listeners here. I worked for a reseller/data aggregator that got acquired by a very large telecom operator, GTT Communications. This is going back, obviously several years, and I was lucky enough to work with GTT for a short time and learned quite a bit. During the time leading up to my working with GTT, we started focusing a lot on the wholesale connectivity space. And a lot of our customers were coming from municipalities and some unfamiliar territories for us. I mean, up until 2015, 2016, we were primarily working with independent ISPs and web-centric companies that had high demand for bandwidth.
Then municipalities started popping up on our radar, power co-ops, and various other institutions that are looking to deploy broadband networks in some hard-to-reach places as you said. And our unique skillset was uniquely positioned to really add a lot of value and help these folks.
While I was working at GTT, I learned a lot, but I realized there was a gaping hole in servicing what is to become a very, very large market of tier three and tier four service providers that are, for the most part, green and unfamiliar with what it’s like to run a network and how to properly connect a remote community to the internet at large. And so I spun off and started Capcon back in 2017, and it’s been straight uphill since then. It’s been fun. We’re helping a lot of people, and we’re having a lot of fun doing it. It’s been great.
Brad Hine:
So, in researching what Capcon does and having a few chats with you now, how did you come to know of these challenges? Obviously, you have experience in this, but I think we could easily say there are probably a lot of listeners out there that may need a little more explanation about what does it mean for an ISP to connect to the internet? They’re the service provider, but where are they connecting and where does that trail lead them? Explain what you’re doing to jump in the loop and help them be more efficient.
Offir Schwartz:
This goes back to my genesis, I would say, in the telecom space. I’m not trying to date myself, but it goes back quite a while to 1999 essentially. So I’ve worked in different segments of the telecom space, co-location for example, IP telephony, wholesale data, retail, the channel, you name it. And so myself and the people on our teams here all have a similar DNA. We’ve been exposed to all these elements of telecom operations over decades. We take all that knowledge, that specific knowledge in that specific industry, and deploy it to help America’s rural broadband operators connect to the internet at large.
And so the types of services that we’re ultimately creating solutions around are your typical services like transport, DDoS mitigation, IP transit, internet exchanges, IPv4 allocations, all of those critical elements that a service provider needs. The rural broadband space is kind of unique because rural broadband operators are confronted with challenges that most other broadband operators in major metros, what we would call tier one or maybe on the elbow of tier one and tier two, are not confronted with. And the first one is just geography, right? Where they’re located. So we’ve amassed contacts at carriers because we’ve primarily bought from them and resold their services to other carriers and vice versa over decades. And so we’ve amassed a database of somewhere over 2,500 carrier operators worldwide. Most of them are in the United States.
And they’re made up of obviously the large incumbent providers, but hundreds of small operators using all different types of media types, whether it be wireless or cable MSO operators, you name it. And so we have at our fingertips knowledge and access to that knowledge that is unique. As we talk about the challenges for broadband operators specifically geography, we know how to bring capacity to these remote communities, leveraging our net of carrier relationships. That’s the first step. And helping them with economies of scale through our buying power.
Because, you know, one of the biggest challenges for these rural broadband operators that we’ve noticed is that they simply just don’t have the economies of scale on their own to compete simply because of the nature of the territories that they’re serving. They’re sparse. There are not as many subscribers as you would find in a tier one or tier two market. And so their buying power is diminished on their own. They need partners to help them play on a level playing field with other operators that would, you know, benefit from having bandwidth readily available to them in a market.
Brad Hine:
Understood. While you are consulting and making the rural broadband operators aware of how to better optimize their network signal, minimize latency, and mitigate traffic and other issues, I heard you say a few things in there. For our audience can you clear up a couple of things about IX transports as opposed to peering gateways and how they play a role in that whole signal?
Offir Schwartz:
Well, boy, that’s a big bag to unpack. I’m going to counter you a little bit. Before we can really talk about IX and transit, those are what in the industry, we consider upstream services, right? But before we can talk about that, there still is a bridge to cross, right? And that’s the transport side, right? That’s, I think, where a lot of rural broadband operators really struggle the most because they still have to connect their networks to these major gateways, these carrier hotels, and IX peering locations that are peppered throughout the country. Knowing where to go and why to go there and how to get there is probably one of the biggest challenges for a broadband operator. So, if it’s okay with you, I’d probably want to start there and kind of work our way back through the network.
Brad Hine:
Absolutely. That’s much more logical. Thank you.
Offir Schwartz:
So the first thing is as a network operator, a rural broadband operator, whether it’s a greenfield deployment or brownfield deployment, where you’re applying for grant funding, whether it be BEAD or RDOF so that you can continue expanding into your territory. It’s really important to have a very firm understanding of what your forecasted subscriber growth looks like over 12 to 36 months. We typically advise operators to look at it on a per-month basis. How many subscribers are you going to add per month? Because what that number is going to do is that’s going to drive the forecast that’ll be used to back into capacity planning. And this is an area that’s oftentimes underrated and overlooked by broadband operators. Not enough attention to detail and effort is being put in upfront to capacity planning.
There are some generic stats that you can use. But what we say is five megabits per second is the peak utilization for your average fiber to the home subscriber throughout the day, right? So a broadband operator is looking to say, okay, well maybe I’m going to add a hundred subscribers per month. That’s as much as 500 megabits per second of bandwidth that they might need month over month that they need to forecast their capacity planning. Does that make sense?
Brad Hine:
Absolutely. Yep. We’re getting the math here.
Offir Schwartz:
Yeah, we’re getting math. And that’s how we typically approach it. We take those numbers and back them into two critical inputs when it comes to capacity planning, which is installation interval and available capacity. There are all sorts of nuances in there in between that, like, where do we connect? Who can we connect with? And where do we connect? Are we going to have an incumbent provider bringing in high-capacity transport? If so, that comes at a cost, right? Because they’re going to have to do the build. Maybe it’s better that we have the broadband operator build some fiber and the transport provider bring some of the fiber to reduce the cost. There are all sorts of nuances around where and how and who to connect with. But as far as what we’re planning for, we’re planning for capacity.
And that installation interval in some cases can be anywhere from, you know, it’s getting better now. The supply chain has loosened up a little bit. But it could be 18 months in some cases for high-capacity export. On the better side, it could be maybe four to six months, depending on the market you’re in and who you’re acquiring that transport from, and what facilities they already have in the area. But we’ve had installations take 18-plus months. If you’re factoring capacity planning and you have to factor 18 months in advance, then you’ve got to double it because you need diversity, right? If one link goes down, you can’t be left on your other links without enough capacity, right? You won’t be able to continue to operate your network.
A funny story, the subscribers don’t really care how you run your network. We’ve kind of learned that the hard way. If my internet is down, there’s a problem. It’s a necessity. And so we’ve heard stories about subscribers calling 911 because their internet is down. So planning for capacity and enough capacity and enough diversity to account for those outages is so important. It cannot be overstated. It cannot be overstated.
So we’re moving back to the network now. If we’re talking about connecting now to major gateways and internet exchanges and peering gateways, there are only about six or eight major gateways in the United States. They’re in major cities, Ashburn, New York, Dallas, Atlanta, Chicago, Miami, and then a few along the west coast as well. And these are carrier hotels. That’s a common term that they go by. This is sort of the center of gravity for the large backbone operators and the true tier-one operators. This is where we typically aim to bring transport back in at the lowest latency possible.
So there are some considerations for the type of transport mode, whether it be optical transport or electrical transport. We were looking at those nuances in choosing the types of transport modes that are available and solving for the best lowest latency connections. Then we’re also solving for cost. There are lots of ways to bake the cake. Rural broadband operators have challenges that, as I said before, other operators would not normally have.
And so really solving for that dollar, that cost per megabyte, that’s a critical number that we’re solving for all the time. How can we inch back down another couple of cents, another couple of dollars, whatever that case may be, what strategies can we deploy? So we’ve got the transport now, and we’re taking it back to these major NFL cities. And one path is going east, and one path is going west. Now we’re at the carrier hotel, so now we have got to connect them somewhere. Who are we connecting them to?
So the internet is made up of two sides. I think a lot of your listeners might really enjoy understanding this because I tend to sort of break it down and boil it down into this easy concept to understand. And that is, there are two sides to the internet. There are only two sides to the internet, and that is eyeballs and content, eyeball provider networks, and content provider networks. Everything in between is just connecting more eyeball networks to more content networks. Eyeball networks are, as you might imagine, folks that are sitting behind their desk and downloading YouTube videos or movies and streaming music, working. The broadband operators that serve those subscribers are considered eyeball networks.
Everybody else is a content network. So Disney +, Hulu, Netflix, you heard of them? They’re a really small streaming company, right? So as you can probably imagine, the biggest bandwidth hogs on broadband operators, and eyeball networks, are Netflix and Disney +. And all those guys are in these big carrier hotels as well. They’re all co-located. All their catching servers and streaming servers are in those buildings as well.
Traditionally speaking, it’s been difficult for rural broadband operators to access what is called peering. So what they’ve done, instead of peering is IP transit which is essentially putting a middleman in between your eyeballs and the content providers.
And those IP transit providers serve a critical need. People might get excited and think that all you must do is connect via peering. But that’s not really feasible. You still need transit. They serve a very, very important purpose in our industry. They are the tier-one backbone providers. The Lumens, the Arelions, the GTT Communications, and the cogent communications of the world, serve a very critical function in our industry. But peering or internet exchange has been commonly overlooked because it’s been very difficult for rural broadband operators to access them for several reasons. We can get into that another podcast or maybe even just later in this conversation.
So these IP transit providers act as a middleman, and they connect you to all the available routes on the internet, including Netflix, Disney, and Hulu. They charge you a per mega rate, right? What we aim to do for the broadband operators that we work with is find the best balance. Because there are solutions, innovative solutions, out there that give broadband operators the ability to compete on a more level playing field by onboarding more than just IP transit, which is how it’s most often done. But also caching servers and internet exchange and peering and all those elements to help them reduce costs, serve their communities better, and just provide an overall better level of service.
So now we’re at the gateways. Once you’re at the gateways, you have choices now. How long am I going to optimize the bandwidth for my network so that I can deliver the highest quality of service to my subscribers? And that’s where the discussion must begin. There are a lot of benefits to Internet exchange. When you’re passing traffic directly to Hulu or directly to Netflix, you’re not paying that middleman, that IP transit provider.
But at the same time, you can’t get out to the whole world of internet routes. You’re only passing the traffic that is destined for that content provider, Netflix. And they’re not going to pass on your traffic beyond their network either. So you need a mix of direct peering in some cases, depending on how much traffic you’ve got, maybe internet exchange, which gives you access via route serve to many content providers and cloud providers and then some IP transit as well, which you’re going to need to get out to the rest of the internet.
Brad Hine:
Understood. Now, you and I spoke initially about the state-run transit provider’s exchange. Does every state have one? And how does that normally work?
Offir Schwartz:
Every state does not have one. Traditionally speaking, a peering exchange is measured by the number of peers that they have in their exchange. A state-bound internet exchange also serves a unique and specific purpose. You don’t typically see the concentration in the density of peers in those exchanges simply because there aren’t enough peers. And there aren’t enough endpoints for those peers. And so they typically gravitate to those major peering gateways, those NFL cities that we talked about.
There’s nothing wrong with participating in those exchanges, but they just can’t be looked at as a silver bullet for solving a connectivity problem or enhancing connectivity. Because often, if you look at those peering, those state exchanges, you’ll see a lot of broadband operators in there passing traffic between each other, but they’re just passing, typically speaking, eyeball-to-eyeball traffic. So it helps keep traffic local so low latency. But it’s not a solution; it’s not an overall solution. It’s a good arrow in the quiver, but not a total solution.
Now there are some interesting conversations that I’ve been having with some folks around open-access networks. And those state exchanges may, soon, play an interesting role. It’s going to be interesting to see how it all plays out. If you do have a state internet exchange that’s predominantly made up of broadband operators, it’s conceivable that an open access operator could essentially deploy their open access solution in that internet exchange. And if they have the connectivity out to metros that they are servicing with their open access solution, any broadband operator in that exchange could potentially participate on that network, which would be a big step forward towards making open access a little bit more available in this nation.
Brad Hine:
Gotcha. Thank you for simplifying all of that and laying that out in that order. So if I’m a rural ISP, how do I know if I’m having latency issues, transfer issues, or transport issues? How do I know if I have a gap there?
Offir Schwartz:
Well, the worst way is to hear it from your customers. So you know, “My video is jittery.” Those types of customer complaints. But a proactive network operator is going to be looking at those metrics using Netflix statistics of various kinds. They’re going to be measuring latency. They’re going to be measuring jitters on their network to these endpoints, to these major content providers. And they’re going to know what’s good and what’s not simply by listening to their customers and watching the stats, watching the numbers.
Brad Hine:
Gotcha. So everyone is talking about all the money that’s out there right now. It was just an announcement this week on how much each state is getting in the infrastructure bill. But speaking of some of the past funding like the CAF, CAF II, RDOF, et cetera, some state-run funding. Did this help or did they work for an area where you focus?
Offir Schwartz:
Oh, absolutely. Yeah, without question. There are a couple of examples I can give you. These guys were not recipients. I don’t want to name names, right? But they did this with municipal funds. They built a network in east Texas. There were incumbent providers like AT&T and Comcast that were unwilling to build into these communities. And I’ve been going out to these communities because we’re based here in Texas. We work with several communities here in Texas. This particular community has just done an exceptional job. They use municipal money, but whether it’s RDOF funding or BEAD money or CAF or any of that, it’s funding. So the source of the funding, I think, is not as important as how effectively you’re able to use that funding.
This particular internet service provider probably had about 5,000 or so subscribers that they were planning on passing. They were estimating, you know, 50, I think, percent take rate just to make the numbers pencil out. And then once they started building the network, the response in that community was overwhelming, upwards of 70- or 80% take rate. And the community has flourished. It’s a totally different place now if you drive out there. You drive through the streets and you can see businesses being erected, street improvements, and a lot of people moving to these communities.
And now, when it’s done properly, this municipality has become the center of gravity for communities all throughout the region, east Texas. So now they are becoming the ISP for other municipalities that are following in their footsteps. And so what they started with was, you know, I don’t remember, it was maybe a couple of gigs of upstream connectivity, transport, and internet. And now they’re upwards of over 200 gigs of capacity to serve this community and the surrounding communities, and they’ve now become the center of gravity for us. So it’s exciting. It’s an exciting time to be in this space. And I digressed a little bit. I hope I didn’t digress too much.
Brad Hine:
Oh, not at all. I think you hit the nail on the head. I was on a couple of calls earlier today, and there was a comment from one of the new interviews we’re trying to set up. The company said, “I don’t know if you’re aware, but this week was pretty significant in terms of the money going into the infrastructure and broadband in the United States.” Everyone on the call started laughing. We’ll never see anything like this in our lifetimes.
Offir Schwartz:
I don’t know, I think we see something bigger yet.
Brad Hine:
Really? Okay.
Offir Schwartz:
I do. Look, a hundred billion between RDOF and BEAD is a big number. But it’s probably only 15%, 20% of what’s needed.
Brad Hine:
Well, you read my mind, because that’s my next question. Do you think it’s enough?
Offir Schwartz:
This is a big country. I mean, there are 60 million people underserved or not served at all. I don’t know if a billion dollars — just do the math on that — or a hundred billion divided by 60 million. So I don’t know if that’s going to be enough money to do it.
Brad Hine:
Right. When we signed you up for this, we asked for stories, stats, examples, samples, and all those things. You’re checking all the boxes here. So thank you so much.
I’m curious at this point. So we have the BEAD money. What’s your estimate of the timeline for the use of the BEAD money? I know there are requirements, but let’s be realistic. In your opinion, how long is it going to take us to get through that bead money and get most of that deployed?
Offir Schwartz:
I think two to five years on the short end and three to seven on the longer side. We’re planning for seven years. There is just going to be a flurry of activity in rural broadband development. I’m already seeing it personally. I don’t know whether it’s because I’m in the industry, but I see a ditch witch on every corner. It seems like orange conduit is going everywhere. Everywhere I go, I see it. There’s like a word for it. I forgot. What is it? Cognitive dissonance, I think. Like, you’re aware of something, you know? You just see more of it.
So there’s a lot of activity, and it’s only going to grow. I think that people just in general need to keep in mind what’s at stake. Because there’s a lot of money out there, and what really is at stake is not the money; it’s the communities. This money really will change people’s lives in ways that one cannot even imagine. Access to the Internet is the frontier for rural communities and for growing economies in rural communities. It’s so important. My hope is that no matter what, whether there is any more money in the future or not, is that we maximize this chance to serve a lot of people and change their lives.
Brad Hine:
Right. Amen. Earlier this week President Biden was comparing this to what happened in the thirties and then later the forties in terms of the electric industry and all the electric networks. So obviously this is a massive opportunity for this country.
I know that you’re passionate about your work with Capcon Networks and everything you do to help these rural communities. Where did you light the fire for technology in your life? Was it always a part of you? Did you just jump into it in high school and college and know? Where did that start in your career?
Offir Schwartz:
It was in college for me. A buddy of mine was doing very well in the co-location world. This was back in ‘99 ish timeframe. And it was intriguing and interesting to me to see the internet in its physical state, right? Because the internet doesn’t exist without a lot of physical infrastructure. I mean, tons of physical infrastructure. And so I was curious and very intrigued by how the internet works, and how it started for the most part.
And I got a firsthand account from a very knowledgeable individual that taught me going back to ARPANET, the very infancy of the internet. He walked me through the entire history, and I was enthralled by it. I wanted to learn as much as I could about it. And that’s what sort of gave me the bug, right? And it’s something that’s kind of unique. I used to live out in LA and the company I worked for had a small data center, and most people have never walked in a data center. Have you?
Brad Hine:
I’ve been to several data centers. Yes. Yeah, they’re very cold, I can tell you that.
Offir Schwartz:
They’re very cold; they’re very loud. And they’re very disorienting. I’ve had somebody pass out in the data center in front of me before. They’re very disorienting if you stay in there, that loud shushing noise from the cooling units and the white floor and the white ceilings. But I’ll never forget. So my in-laws were in town, and I invited them to our office. I gave them a data center tour. And the only way that I could explain it to them was, “You’re looking at the internet. This is the internet right here. Do you see all those cables and all those risers and those black fiber optic cables? This is where the connectivity comes in, and this is where the servers are that give you all your content.”
And it just kind of clicked for them. That’s really sort of how I got into it and stayed passionate about it. When this whole rural broadband, I mean, it’s been an issue for a long time. But when so much attention was put towards it, especially post covid or during Covid. That really shined a spotlight on it. I knew I was in the right place.
Brad Hine:
Wow. That’s very cool. I know from working in the industry now for over 15 years and working with a lot of ISPs, traditional telecoms, and cable operators, there’s always that time when someone says, “You want to see our data center? You want to see our server room.”
And it’s fun to walk through and kind of compare them historically and see how they’re set up and how detailed they are.
Offir Schwartz:
For listeners, it’s called “Tubes”. I forgot the author’s name, but I’m sure if they just typed or Googled, “Tubes Internet”, they will find it. It’s a short read. It’s an excellent read for anybody that wants to understand how when I hit enter on my keyboard when I’m sending a picture to my grandma in five seconds it shows up on the other side of the world.
Brad Hine:
Aha. Very cool. Well, that’s a great shout-out. Everyone go to Amazon or the Internet and look up “Tubes”.
As we wind down our episode today, I want to tell you that I appreciate you walking us through that timeline and the different pieces of the network, how it’s set up, and how these rural folks are connecting. We have this sequence; this part of our show is called the Back to the Future Question. And it’s very simple. If you got in your time machine, your DeLorean, and went back in time, if you were going to try to make your life today a little easier, give yourself kind of a leg up by whispering in your ear, 10, 20 years ago and how to simplify all this. What kind of advice would you give yourself back then?
Offir Schwartz:
Well, if, if I really wanted to make my life easier, I would’ve invested in Amazon 20 years ago.
But in this context for this podcast today, I would’ve said, stay the course. And that’s exactly what I did. You know what I mean? I stayed on the course. I stayed with it through the good times and the bad times. And trust me, there have been a lot of bad times in the telecom industry. I’ve stayed true to my passion, which is learning and teaching others about what I know. I’ve always done this.
Let’s keep in mind what’s at stake here. And for me, it’s never about the money. It’s never about the vendors and the products and the services there’s always one clear destiny or endpoint. In this case, it’s people’s lives. And in the context of what we do, it’s connecting them to the internet to give them a better quality of life. And that’s what we do. That’s what I’ve sort of stayed true to all the time.
Brad Hine:
Wise words. Well, I think it’s going to wrap up another episode of the Broadband Bunch. I want to thank all of our listeners and to thank Offir Schwartz for joining. If anyone wants to reach out to you or Capcon Networks, how could they do that?
Offir Schwartz:
Offir@Capconnetworks.com. They can also go to our website Capconnetworks.com and there’s a “Contact Us” button. I’m always available as well, so you can reach me directly just by my email address Offir@Capconnetworks.com. That’s the easiest way to get me.
Brad Hine:
Perfect. And for all our listeners out there, check out our site, broadbandbunch.com for weekly episodes and resources. Contact us to share your broadband stories with us. From everyone at the Broadband Bunch to all of you, thanks again. Have a great week.
