Bringing a fiber network to Northern Alabama - ETI
X

Want to take a Self-Guided tour?




February 1, 2023

Bringing a fiber network to Northern Alabama

The following transcript has been edited for length and readability. Listen to the entire discussion here on The Broadband Bunch. The Broadband Bunch is sponsored by ETI Software.

Pete Pizzutillo:

Hello and welcome to another episode of the Broadband Bunch. My name is Pete Pizzutillo and we are at Calix Connections 2022 at the Wynn Resort in Las Vegas. And I am joined by Mark Freeman. He is the Manager of Network Operations for Cullman Electric Cooperative. And Mark, thanks for joining the show today.

Mark Freeman:

Today. Yeah, thanks, Pete. Appreciate the opportunity.

Pete Pizzutillo:

And there’s a lot going on here. And before we get into the event and some of the things that you’re seeing, maybe just a little bit of backdrop on who you are and a little bit more about Cullman.

Mark Freeman:

Sure, yeah. So, my name’s Mark Freeman, Manager of Network Operations for Sprout Fiber Internet. Sprout Fiber Internet is a subsidiary of Cullman Electric Cooperative and it is the name of our fiber broadband project that we are doing. That project started… Our first subscriber was in January of 2021, where we turned up our first subscriber and right now we have hit about 4,400 subscribers.

Pete Pizzutillo:

Good lord.

Mark Freeman:

So it’s been a great journey just to see it going through. So I manage the network side. So I designed the Sprout Fiber Internet, the architecture that we have, and got it all turned up. And I have a team of four network technicians that report to me and we support the tier two calls of the network and also all the service calls. And then my counterpart, Bonnie Baty, is the manager of customer service, so she kind of handles the scheduling and the marketing aspect and the commercial sales of that project.

So I’ve been with Cullman now for about two and a half years. They hired me for the network experience that I brought. I previously turned up another co-op in Mississippi at Elkhorn County Electric. So I was their consultant for them and helped them get their network online. And then I transitioned from there over to Cullman Electric because they were in need of a network engineer.

And I’ve been there for about two and a half years now. It’s been great. And phase one of our projects. The purpose of a co-op is to service members, right?

Pete Pizzutillo:

Right.

Mark Freeman:

And co-ops were created because of a need and that need was power, right? So co-op Cullman Electric was created to serve a need of meeting power for our members. And Sprout Fiber Internet came along because we saw the need for broadband internet in our community, and our members were desperately needing that. So the board decided to move forward with that and that’s how Sprout came about was, again, meeting the need of our members and creating that broadband subsidiary.

Pete Pizzutillo:

So about the need, was it that the area was underserved or just not fairly served by some of the providers?

Mark Freeman:

Completely underserved.

Pete Pizzutillo:

Okay.

Mark Freeman:

So inside Cullman, there’s a municipality, Cullman Power Board. So Sprout doesn’t actually serve that. So Cullman Electric serves around the city. So you have the municipality inside of Cullman, and then since we’re a rural cooperative, we serve everything around Cullman. So there’s no problem getting internet in the city, or if you are a member around the city, but when you get to the edge of our network, we have members that have no internet at all. So I was just talking to another person today about, we had a comment on our Facebook group where somebody had a 1.25 meg download and a 0.15 upload, and they live near Bankhead National Forest on the very edge of our network that we just turned up. And she went from that to 941 meg down and 941 meg up.

Pete Pizzutillo:

Wow.

Mark Freeman:

So those are the kind of changes that we’re doing in Alabama. And there are people that don’t even have cell phone service, they can’t do hotspots, they have nothing. And we’re changing people’s lives to be able to work from home, to allow their kids to do school. So that’s one of the joys of what I do is just seeing the lives that we’re changing and knowing that we’re impacting people’s lives. It kind of makes up for all the crazy late hours and outages that we have and just the stress when you’re seeing those kinds of stories. Our marketing people try to share those with my team, because sometimes we’re in the grind, right, and we just see all the problems and the issues. And it’s always good to see those positive things that people are posting on social media about how we’re affecting them.

Pete Pizzutillo:

And I don’t think you’ll fully appreciate it for decades, right? I mean, because you’re really starting from the young folks that need to learn and to folks that can stay in their home a little bit longer because they have some more independence. So thank you for doing that. That’s a great service to your community. How do you go from one to 4,000 in under two years? What’s the secret of your success there?

Mark Freeman:

Well, I just met another co-op today. They went from zero to 32,000 in four years. They were doing, I mean, I was just like, I floored, I couldn’t believe it. But what we did is I’ve seen a lot of co-ops go out and they just right out the gate start out with 20, 30-something installs a day. And from a marketing perspective, when you start up, people are going to ask those subscribers, how do you like Sprout, right?

Pete Pizzutillo:

Right.

Mark Freeman:

So word of mouth, as you probably know, is the biggest thing. What we didn’t want was to jump out of the gate with a high turn-up, have a bad experience, and then people tell others about that bad experience to others. So what we did, we took the approach of, we started out with four installs a day, just one in each. We have four-time slots that we do. So we did a very slow rollout. We were deploying a brand new billing system. We used GLDS, which has been completely awesome with the automation process that they have. And so, we worked, I mean think about it, you’re turning up a completely brand new internet company from scratch, and you’re a utility company. You have no idea what the internet it is, right?

Pete Pizzutillo:

Right.

Mark Freeman:

So I mean it’s a lot of hurdles, I mean even for me, with 21 years of experience in telecommunications, there’s still new stuff that you’re dealing with every day, right? And then when we were launching in the early months, my team hated it, or the team I was working with because the procedures just felt like I was redoing procedures every day. Okay, hey we’re going to do this now when we do a turn-up and then the next day, it’s like okay, well we need to modify that and do this. So I was constantly updating procedure documents because you don’t know what you don’t know.

Pete Pizzutillo:

Exactly.

Mark Freeman:

Because every day it’s a new challenge, or a customer or subscriber has a new situation and you have to modify it and adapt it. And that’s what I love about the job it’s something different every single day. It’s a new challenge, it’s a new struggle, and I’m never bored. That’s what I love about it is this, you’re always accomplishing or tackling a new task, a new objective, and you’re changing, and that’s one of the main reasons why I love what I do.

Pete Pizzutillo:

But if you take a step back and look at traditionally the criticism around utilities and maybe co-ops that they haven’t been very competitive, right? They haven’t been able to move fast and be nimble. So what you just described is kind of an agile development of your network, your customer service, and your business model. So what was that cultural clash like or how did you guys get past that? Or is that not something that you experienced?

Mark Freeman:

I mean, one thing to note too is that we’re completely 1% self-sufficient. So we’ve taken out loans to pay for our project. So that right there is the number one reason why we’re not going gangbusters out of the block is that we are spending our member’s money and we have to respect that and spend it in a way that honors them without putting the co-op in debt. So that’s the biggest thing is that we’re not a typical internet company. We are owned by our members. So we have to spend, and deploy, and think about how we do it from the mindset of our board and our members to accurately and proficiently spend that money in a way that makes a project move and make the money without just going completely in debt right out the gate.

We’ve applied for a lot of fundal grants and just have not been receiving any. That’s been another challenge. But going back to your question though is we did that small rollout at the very beginning, and I highly encourage other co-ops, people getting into to do that, because it allows you to get all those procedures in place so that you can ramp up to where we’re at now, which is around 20 to 24 installs a day, and we’re over six weeks backed up right now in orders.

Pete Pizzutillo:

Wow.

Mark Freeman:

And we can talk about it in a minute, but some of that is because of supply chain issues that we’re having. But definitely just getting those procedures in place as you’re rolling out is the most important thing that when you go out of the gate full blast like that, you don’t have time to create this. I mean you’re literally just going at the seat of your pants just rolling it out. So it’s very challenging.

Pete Pizzutillo:

Yeah, I think there are also too many variables to figure out what’s working, and what’s not working when you’re at a certain scale. So that is a really interesting crawl, walk, run kind of approach. And I think to your point about word of mouth, especially in communities, tight-knit communities, right? I mean, you don’t get a second shot.

Mark Freeman:

Exactly.

Pete Pizzutillo:

So what challenges do you see next in your next phase of growth that you’re interested in and eager to tackle?

Mark Freeman:

So right now we have just, we’re a utility company, right? So even though that we’re selling broadband internet, the whole reason that Cullman Electric got in, one of the other major reasons we got into it is, we wanted to provide fiber connectivity to our downline devices to make our network more effective and more productive for our members to bring down their costs. So phase one of our project was interconnecting 13 substations. So we have 13 substations that have fiber huts there. We have a total of 17 substations, but some of them have some legs going off to them. But on the 100 gig ring we have, we have 13 fiber huts that are interconnected with the 100 gig ring. And that was our phase, and we were turning up subscribers as we were building that backbone ring. And then now we’re in phase two, which is we’re just turning up zones, like little pockets of the area around that ring. And that’s kind of where we’re at right now.

So some of the challenges that we have, now that we have things up, as you probably know, the biggest issue we have right now is just material. So we would love to fully roll out 24 or more installs a day, but we can’t because some of the material that we need on the drop side, goes from the utility pole to the home, there’s a shortage of that. And we’re having to spend a lot more money and man hours to makeshift something that does work to overcompensate where we’re not getting our material. And that’s been a huge tackle. So it’s either we have to do it slower or we don’t do it at all.

So we’re having to cut back, but then, as you know, we have every subscriber screaming at us, but it is like we can’t get the internet to their house faster. So it’s a difficult situation and we actually had to put out a social media post that we’re having to push people’s back. And you don’t want to get that email. We’re like, oh hey, you’re going live in October. Oh, now you’re December. So it’s a difficult thing, but that’s probably the biggest challenge is what I would recommend is, it’s hard because you want to stock the material, but to stock that kind of material, you have to have a warehouse and you got to have, I mean, we already have a large warehouse full of electric equipment, and I’m shoving a lot of my electronics in there. Then we have another huge warehouse down the road that we have fiber and it’s got a complete lot just slammed full of equipment, and we need another warehouse now because at today’s day and age, if you want to run a productive environment, you got to have that material to plan for these shortages and that requires space.

And that’s the issue that we’re having right now we’re having to go look for more space to store all the material in anticipation of it running out. And it’s a lot more money too because you have to manage the inventory and spend all the money on fiber. And so, that’s the biggest thing. And that’s where we got a pickle is we didn’t stock enough of that drop material because we didn’t see this coming and that’s what got us and one of them is not having enough [inaudible 00:13:20]. So for me, on the electric side, on the electronic side, what I’m doing is I’m stocking six months of equipment out. So if something does happen, I have six months of O&Ts, I have six months of Calix RGs available to last me six months if for some reason I’m not getting that ahead of time.

Pete Pizzutillo:

Well, you couldn’t anticipate that, I mean, I guess from lessons learned, you know have the benefit of building a broadband network from scratch, kind of thinking about what the future looks like and making designing for the future. Somebody who’s going through the same kind of process right now, what should they be thinking about in terms of scale, affordability, and resiliency?

Mark Freeman:

Man, it’s a whole different ballgame now. I mean, I’m trying to buy some new core switches right now, and just to get two of them, it’s over a year’s wait time. So even if you’re thinking of getting into broadband internet, you need to hire a network guy, get that network infrastructure designed, and get that ordered equipment. I was talking to another co-op, they’re starting up, and their board told them that they wanted to get up by December, and I said you can forget that. It’s like you’re not getting a core router.

Pete Pizzutillo:

Which December?

Mark Freeman:

You’re not getting a core router in two months to go install it. And that’s the biggest thing is getting that equipment ordered. For us, one of the challenges we’ve had is home pass records. You would think that as a utility company, we would know everybody’s service address, but utility companies go off what they call map numbers, or for us, it’s an identifier on a grid of where that person lived. So when there’s an issue, that lineman goes to this map number, they don’t really care what the address is, right? So here we are, as a service provider, trying to deploy the internet, and guess what? You can’t deploy the internet if you don’t have that home’s pass address. So we’ve been having to go back and clean up 80-something years of bad data on the utility side and put home’s pass record, update all that. And it takes a very long time to clean all that, because okay, hey, is this meter serving a barn? Is it a serviceable address? So a lot of it is you’re going into Google Maps and you’re looking to see where this pin is. Is this latitude to longitude? What is actually there? And it’s a manual task with a lot of time.

But having good partners, like GLDS and the other partner Logicom, partnered with us to help us seam line that as well as possible to make it more efficient, to get that and to get that into the billing system so that people can sign up. GLDS has an awesome home import utility tool that we use to get addresses imported into that, so people can sign up really quickly. So that’s the biggest key, one of the biggest challenges we’ve had. And a lot of times, co-ops, don’t think about it until after, it hit us after we were already installed. You know need to be thinking of that ahead of time. Do you want that? But people keep asking what your take rate is, right? If you don’t have an accurate home pass record, your take rate is complete garbage. So we haven’t been able to tell take rates, we’ve been just going off of active subscribers, but now that we’ve gone back and cleaned up our data, we’re now able to start seeing more accurate take rates, which is what the board and everybody want to see.

Pete Pizzutillo:

Right. So preparing the ground both on the supply chain side and on the data side, while you’re thinking about your planning, there are things that you all can be doing to make sure that once you have a plan approved and funded you can go. Right?

Mark Freeman:

Yeah. Maybe just feedback on one other question that you had, on the previous one. One of the other things that I always tell people when they’re getting into broadband internet is getting good partners. Kind of just feeding back on that. And my project would not be successful without having the key partners that I have, and that’s probably one of the most important is being able to trust that partner and to know that the product that you’re deploying in your field is going to have maximum uptime, and also when there is an issue that you get it resolved.

And one of the biggest advantages I have is, again, is with my partners working together and having GLDS and Calix, we’ve just turned up an API that allows them two to talk to each other. So they can push information and pull information. So that when my subscribers are getting turned up, in real-time, that information from my billing system, GLDS, is pushed automatically into the supported cloud. So if that subscriber ever calls in, the team member that I manage can go into the supported cloud, search on that account number, and their name, and get real-time information about what’s going on. So having them, when there’s an issue, get on the phone with GLDS, or Calix, or any of the major partners that we have, and just being able to rely on them and to get you past those difficulties.

And that’s what really why I’m here at the show right now to kind of get it back to your question Calix was able to overcome a hardware shortage that we had and that opened the door to us using Calix because we ran out of inventory of our residential gateways and we were about to stop shop after we already stopped shop a year ago due to construction issues. We had to stop construction for four months, and it was brutal. And that pushed our project back. And here I am over the network and oh my gosh we don’t have any RGs, guess what?

You can’t turn up a subscriber without a wireless gateway. So we had just opened the contract with Calix for using them for our commercial routers, and my sales rep, Lily, came in, saved the day, and within a month, I had over 2000 RGs sitting in my warehouse for deployment. And ever since then, just knowing that you have a partner that can have the equipment and have it ready for you to use when you need it and order it, is a huge, especially in today’s market and the way the material’s going nowadays, that was a game changer for us was knowing that when I place an order it’s going to be here when they tell me it’s going to be here.

Pete Pizzutillo:

Yeah, no, that’s a great testament, and you see that a lot here at this event. The ecosystem is pretty strong and it just keeps growing. So you’ve been to Calix before. This is the second year. What are you seeing that’s of interest to you?

Mark Freeman:

Sure. So what really drew me to Calix Connections is when I was here last year, it was really a game changer for me. I’m a small cooperative, right? So I consider myself a very small service provider compared to the tier ones, the tier twos, that’s in today, right? And we only have around 4,000 subscribers right now. And for a partner like Calix to come and step beside us and to help us, that was what blew me away they made us feel like we were a tier one. Where I’m meeting with their VPs in a room with just us and another co-op, and they’re asking us what we want in their products.

Pete Pizzutillo:

Yeah.

Mark Freeman:

And I’m like, do you guys have the right people in this meeting? You know you’re talking to a small co-op in Alabama, right? Do you really want my info? But that really was a game-changer for me. And they’re listening to small co-ops in Alabama and Tennessee, like Forked Deer, and they’re asking for our input on how they want their integrations with the supported cloud and GLDS, and what they envision us to look like. I just never had that before, and it was a real game-changer for me. I’m just, okay, these people really care about us and they want us to be successful, and that’s just a great feeling that you know you have somebody like that beside you that is going to stand there and go to battle with you to overcome these big giants that are out there in the field right now trying to take over our markets.

Pete Pizzutillo:

Yeah, a hundred percent. I mean, having a gentle giant come and take you by the wrist and help you say it’s going to be okay. They’re really good at capturing the voice of the customer.

Pete Pizzutillo:

Mark, thanks for sharing the story about Sprout, and good luck with the rest of your rollout in phase two, and maybe we’ll get you back on the show and you can tell us about how things look in a year.

Mark Freeman:

Appreciate it. Most definitely. Thank you for the opportunity.

Pete Pizzutillo:

Thanks man.